{"cells":[{"cell_type":"markdown","id":"aff22554","metadata":{"id":"aff22554"},"source":["The following additional libraries are needed to run this\n","notebook. Note that running on Colab is experimental, please report a Github\n","issue if you have any problem."]},{"cell_type":"markdown","metadata":{"origin_pos":0,"id":"f8ecbb15"},"source":["# 权重衰减\n",":label:`sec_weight_decay`\n","\n","前一节我们描述了过拟合的问题，本节我们将介绍一些正则化模型的技术。\n","我们总是可以通过去收集更多的训练数据来缓解过拟合。\n","但这可能成本很高，耗时颇多，或者完全超出我们的控制，因而在短期内不可能做到。\n","假设我们已经拥有尽可能多的高质量数据，我们便可以将重点放在正则化技术上。\n","\n","回想一下，在多项式回归的例子（ :numref:`sec_model_selection`）中，\n","我们可以通过调整拟合多项式的阶数来限制模型的容量。\n","实际上，限制特征的数量是缓解过拟合的一种常用技术。\n","然而，简单地丢弃特征对于这项工作来说可能过于生硬。\n","我们继续思考多项式回归的例子，考虑高维输入可能发生的情况。\n","多项式对多变量数据的自然扩展称为*单项式*（monomials），\n","也可以说是变量幂的乘积。\n","单项式的阶数是幂的和。\n","例如，$x_1^2 x_2$和$x_3 x_5^2$都是3次单项式。\n","\n","注意，随着阶数$d$的增长，带有阶数$d$的项数迅速增加。 \n","给定$k$个变量，阶数为$d$的项的个数为\n","${k - 1 + d} \\choose {k - 1}$，即$C^{k-1}_{k-1+d} = \\frac{(k-1+d)!}{(d)!(k-1)!}$。\n","因此即使是阶数上的微小变化，比如从$2$到$3$，也会显著增加我们模型的复杂性。\n","仅仅通过简单的限制特征数量（在多项式回归中体现为限制阶数），可能仍然使模型在过简单和过复杂中徘徊，\n","我们需要一个更细粒度的工具来调整函数的复杂性，使其达到一个合适的平衡位置。\n","## 范数与权重衰减\n","\n","在 :numref:`subsec_lin-algebra-norms`中，\n","我们已经描述了$L_2$范数和$L_1$范数，\n","它们是更为一般的$L_p$范数的特殊情况。\n","(~~权重衰减是最广泛使用的正则化的技术之一~~)\n","在训练参数化机器学习模型时，\n","*权重衰减*（weight decay）是最广泛使用的正则化的技术之一，\n","它通常也被称为$L_2$*正则化*。\n","这项技术通过函数与零的距离来衡量函数的复杂度，\n","因为在所有函数$f$中，函数$f = 0$（所有输入都得到值$0$）\n","在某种意义上是最简单的。\n","但是我们应该如何精确地测量一个函数和零之间的距离呢？\n","没有一个正确的答案。\n","事实上，函数分析和巴拿赫空间理论的研究，都在致力于回答这个问题。\n","\n","一种简单的方法是通过线性函数\n","$f(\\mathbf{x}) = \\mathbf{w}^\\top \\mathbf{x}$\n","中的权重向量的某个范数来度量其复杂性，\n","例如$\\| \\mathbf{w} \\|^2$。\n","要保证权重向量比较小，\n","最常用方法是将其范数作为惩罚项加到最小化损失的问题中。\n","将原来的训练目标*最小化训练标签上的预测损失*，\n","调整为*最小化预测损失和惩罚项之和*。\n","现在，如果我们的权重向量增长的太大，\n","我们的学习算法可能会更集中于最小化权重范数$\\| \\mathbf{w} \\|^2$。\n","这正是我们想要的。\n","让我们回顾一下 :numref:`sec_linear_regression`中的线性回归例子。\n","我们的损失由下式给出：\n","\n","$$L(\\mathbf{w}, b) = \\frac{1}{n}\\sum_{i=1}^n \\frac{1}{2}\\left(\\mathbf{w}^\\top \\mathbf{x}^{(i)} + b - y^{(i)}\\right)^2.$$\n","\n","回想一下，$\\mathbf{x}^{(i)}$是样本$i$的特征，\n","$y^{(i)}$是样本$i$的标签，\n","$(\\mathbf{w}, b)$是权重和偏置参数。\n","为了惩罚权重向量的大小，\n","我们必须以某种方式在损失函数中添加$\\| \\mathbf{w} \\|^2$，\n","但是模型应该如何平衡这个新的额外惩罚的损失？\n","实际上，我们通过*正则化常数*$\\lambda$来描述这种权衡，\n","这是一个非负超参数，我们使用验证数据拟合：\n","\n","$$L(\\mathbf{w}, b) + \\frac{\\lambda}{2} \\|\\mathbf{w}\\|^2,$$\n","\n","对于$\\lambda = 0$，我们恢复了原来的损失函数。\n","对于$\\lambda > 0$，我们限制$\\| \\mathbf{w} \\|$的大小。\n","这里我们仍然除以$2$：当我们取一个二次函数的导数时，\n","$2$和$1/2$会抵消，以确保更新表达式看起来既漂亮又简单。\n","你可能会想知道为什么我们使用平方范数而不是标准范数（即欧几里得距离）？\n","我们这样做是为了便于计算。\n","通过平方$L_2$范数，我们去掉平方根，留下权重向量每个分量的平方和。\n","这使得惩罚的导数很容易计算：导数的和等于和的导数。\n","\n","此外，你可能会问为什么我们首先使用$L_2$范数，而不是$L_1$范数。\n","事实上，这个选择在整个统计领域中都是有效的和受欢迎的。\n","$L_2$正则化线性模型构成经典的*岭回归*（ridge regression）算法，\n","$L_1$正则化线性回归是统计学中类似的基本模型，\n","通常被称为*套索回归*（lasso regression）。\n","使用$L_2$范数的一个原因是它对权重向量的大分量施加了巨大的惩罚。\n","这使得我们的学习算法偏向于在大量特征上均匀分布权重的模型。\n","在实践中，这可能使它们对单个变量中的观测误差更为稳定。\n","相比之下，$L_1$惩罚会导致模型将权重集中在一小部分特征上，\n","而将其他权重清除为零。\n","这称为*特征选择*（feature selection），这可能是其他场景下需要的。\n","\n","使用与 :eqref:`eq_linreg_batch_update`中的相同符号，\n","$L_2$正则化回归的小批量随机梯度下降更新如下式：\n","\n","$$\n","\\begin{aligned}\n","\\mathbf{w} & \\leftarrow \\left(1- \\eta\\lambda \\right) \\mathbf{w} - \\frac{\\eta}{|\\mathcal{B}|} \\sum_{i \\in \\mathcal{B}} \\mathbf{x}^{(i)} \\left(\\mathbf{w}^\\top \\mathbf{x}^{(i)} + b - y^{(i)}\\right).\n","\\end{aligned}\n","$$\n","\n","根据之前章节所讲的，我们根据估计值与观测值之间的差异来更新$\\mathbf{w}$。\n","然而，我们同时也在试图将$\\mathbf{w}$的大小缩小到零。\n","这就是为什么这种方法有时被称为*权重衰减*。\n","我们仅考虑惩罚项，优化算法在训练的每一步*衰减*权重。\n","与特征选择相比，权重衰减为我们提供了一种连续的机制来调整函数的复杂度。\n","较小的$\\lambda$值对应较少约束的$\\mathbf{w}$，\n","而较大的$\\lambda$值对$\\mathbf{w}$的约束更大。\n","\n","是否对相应的偏置$b^2$进行惩罚在不同的实践中会有所不同，\n","在神经网络的不同层中也会有所不同。\n","通常，网络输出层的偏置项不会被正则化。\n","\n","## 高维线性回归\n","\n","我们通过一个简单的例子来演示权重衰减。\n"],"id":"f8ecbb15"},{"cell_type":"code","execution_count":null,"metadata":{"execution":{"iopub.execute_input":"2022-07-31T02:22:47.330465Z","iopub.status.busy":"2022-07-31T02:22:47.329932Z","iopub.status.idle":"2022-07-31T02:22:49.309063Z","shell.execute_reply":"2022-07-31T02:22:49.308346Z"},"origin_pos":2,"tab":["pytorch"],"id":"cc703e75"},"outputs":[],"source":["%matplotlib inline\n","import torch\n","from torch import nn\n","from d2l import torch as d2l"],"id":"cc703e75"},{"cell_type":"markdown","metadata":{"origin_pos":4,"id":"f836b7d8"},"source":["首先，我们[**像以前一样生成一些数据**]，生成公式如下：\n","\n","(**$$y = 0.05 + \\sum_{i = 1}^d 0.01 x_i + \\epsilon \\text{ where }\n","\\epsilon \\sim \\mathcal{N}(0, 0.01^2).$$**)\n","\n","我们选择标签是关于输入的线性函数。\n","标签同时被均值为0，标准差为0.01高斯噪声破坏。\n","为了使过拟合的效果更加明显，我们可以将问题的维数增加到$d = 200$，\n","并使用一个只包含20个样本的小训练集。\n"],"id":"f836b7d8"},{"cell_type":"code","execution_count":null,"metadata":{"execution":{"iopub.execute_input":"2022-07-31T02:22:49.312954Z","iopub.status.busy":"2022-07-31T02:22:49.312444Z","iopub.status.idle":"2022-07-31T02:22:49.319046Z","shell.execute_reply":"2022-07-31T02:22:49.318390Z"},"origin_pos":5,"tab":["pytorch"],"id":"b21a3d9c"},"outputs":[],"source":["n_train, n_test, num_inputs, batch_size = 20, 100, 200, 5\n","true_w, true_b = torch.ones((num_inputs, 1)) * 0.01, 0.05\n","train_data = d2l.synthetic_data(true_w, true_b, n_train)\n","train_iter = d2l.load_array(train_data, batch_size)\n","test_data = d2l.synthetic_data(true_w, true_b, n_test)\n","test_iter = d2l.load_array(test_data, batch_size, is_train=False)"],"id":"b21a3d9c"},{"cell_type":"markdown","metadata":{"origin_pos":6,"id":"93ed27b1"},"source":["## 从零开始实现\n","\n","下面我们将从头开始实现权重衰减，只需将$L_2$的平方惩罚添加到原始目标函数中。\n","\n","### [**初始化模型参数**]\n","\n","首先，我们将定义一个函数来随机初始化模型参数。\n"],"id":"93ed27b1"},{"cell_type":"code","execution_count":null,"metadata":{"execution":{"iopub.execute_input":"2022-07-31T02:22:49.323345Z","iopub.status.busy":"2022-07-31T02:22:49.322872Z","iopub.status.idle":"2022-07-31T02:22:49.327215Z","shell.execute_reply":"2022-07-31T02:22:49.326404Z"},"origin_pos":8,"tab":["pytorch"],"id":"5e5a8e3a"},"outputs":[],"source":["def init_params():\n","    w = torch.normal(0, 1, size=(num_inputs, 1), requires_grad=True)\n","    b = torch.zeros(1, requires_grad=True)\n","    return [w, b]"],"id":"5e5a8e3a"},{"cell_type":"markdown","metadata":{"origin_pos":10,"id":"0a6c622e"},"source":["### (**定义$L_2$范数惩罚**)\n","\n","实现这一惩罚最方便的方法是对所有项求平方后并将它们求和。\n"],"id":"0a6c622e"},{"cell_type":"code","execution_count":null,"metadata":{"execution":{"iopub.execute_input":"2022-07-31T02:22:49.331227Z","iopub.status.busy":"2022-07-31T02:22:49.330783Z","iopub.status.idle":"2022-07-31T02:22:49.334599Z","shell.execute_reply":"2022-07-31T02:22:49.333840Z"},"origin_pos":12,"tab":["pytorch"],"id":"61a7fa99"},"outputs":[],"source":["def l2_penalty(w):\n","    return torch.sum(w.pow(2)) / 2"],"id":"61a7fa99"},{"cell_type":"markdown","metadata":{"origin_pos":14,"id":"47b5816b"},"source":["### [**定义训练代码实现**]\n","\n","下面的代码将模型拟合训练数据集，并在测试数据集上进行评估。\n","从 :numref:`chap_linear`以来，线性网络和平方损失没有变化，\n","所以我们通过`d2l.linreg`和`d2l.squared_loss`导入它们。\n","唯一的变化是损失现在包括了惩罚项。\n"],"id":"47b5816b"},{"cell_type":"code","execution_count":null,"metadata":{"execution":{"iopub.execute_input":"2022-07-31T02:22:49.338662Z","iopub.status.busy":"2022-07-31T02:22:49.338197Z","iopub.status.idle":"2022-07-31T02:22:49.344902Z","shell.execute_reply":"2022-07-31T02:22:49.344113Z"},"origin_pos":16,"tab":["pytorch"],"id":"a559f80b"},"outputs":[],"source":["def train(lambd):\n","    w, b = init_params()\n","    net, loss = lambda X: d2l.linreg(X, w, b), d2l.squared_loss\n","    num_epochs, lr = 100, 0.003\n","    animator = d2l.Animator(xlabel='epochs', ylabel='loss', yscale='log',\n","                            xlim=[5, num_epochs], legend=['train', 'test'])\n","    for epoch in range(num_epochs):\n","        for X, y in train_iter:\n","            # 增加了L2范数惩罚项，\n","            # 广播机制使l2_penalty(w)成为一个长度为batch_size的向量\n","            l = loss(net(X), y) + lambd * l2_penalty(w)\n","            l.sum().backward()\n","            d2l.sgd([w, b], lr, batch_size)\n","        if (epoch + 1) % 5 == 0:\n","            animator.add(epoch + 1, (d2l.evaluate_loss(net, train_iter, loss),\n","                                     d2l.evaluate_loss(net, test_iter, loss)))\n","    print('w的L2范数是：', torch.norm(w).item())"],"id":"a559f80b"},{"cell_type":"markdown","metadata":{"origin_pos":18,"id":"72256efc"},"source":["### [**忽略正则化直接训练**]\n","\n","我们现在用`lambd = 0`禁用权重衰减后运行这个代码。\n","注意，这里训练误差有了减少，但测试误差没有减少，\n","这意味着出现了严重的过拟合。\n"],"id":"72256efc"},{"cell_type":"code","execution_count":null,"metadata":{"execution":{"iopub.execute_input":"2022-07-31T02:22:49.348006Z","iopub.status.busy":"2022-07-31T02:22:49.347416Z","iopub.status.idle":"2022-07-31T02:22:55.832837Z","shell.execute_reply":"2022-07-31T02:22:55.832068Z"},"origin_pos":19,"tab":["pytorch"],"id":"045cb210","outputId":"7d0c9c39-cae7-4882-8415-f1c012e36c27"},"outputs":[{"name":"stdout","output_type":"stream","text":["w的L2范数是： 12.791678428649902\n"]},{"data":{"image/svg+xml":["<?xml version=\"1.0\" encoding=\"utf-8\" standalone=\"no\"?>\n","<!DOCTYPE svg PUBLIC \"-//W3C//DTD SVG 1.1//EN\"\n","  \"http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd\">\n","<svg xmlns:xlink=\"http://www.w3.org/1999/xlink\" width=\"263.421875pt\" height=\"180.65625pt\" viewBox=\"0 0 263.421875 180.65625\" xmlns=\"http://www.w3.org/2000/svg\" version=\"1.1\">\n"," <metadata>\n","  <rdf:RDF xmlns:dc=\"http://purl.org/dc/elements/1.1/\" xmlns:cc=\"http://creativecommons.org/ns#\" xmlns:rdf=\"http://www.w3.org/1999/02/22-rdf-syntax-ns#\">\n","   <cc:Work>\n","    <dc:type rdf:resource=\"http://purl.org/dc/dcmitype/StillImage\"/>\n","    <dc:date>2022-07-31T02:22:55.759832</dc:date>\n","    <dc:format>image/svg+xml</dc:format>\n","    <dc:creator>\n","     <cc:Agent>\n","      <dc:title>Matplotlib v3.5.1, https://matplotlib.org/</dc:title>\n","     </cc:Agent>\n","    </dc:creator>\n","   </cc:Work>\n","  </rdf:RDF>\n"," </metadata>\n"," <defs>\n","  <style type=\"text/css\">*{stroke-linejoin: round; stroke-linecap: butt}</style>\n"," </defs>\n"," <g id=\"figure_1\">\n","  <g id=\"patch_1\">\n","   <path d=\"M 0 180.65625 \n","L 263.421875 180.65625 \n","L 263.421875 0 \n","L 0 0 \n","L 0 180.65625 \n","z\n","\" style=\"fill: none\"/>\n","  </g>\n","  <g id=\"axes_1\">\n","   <g id=\"patch_2\">\n","    <path d=\"M 51.378125 143.1 \n","L 246.678125 143.1 \n","L 246.678125 7.2 \n","L 51.378125 7.2 \n","z\n","\" style=\"fill: #ffffff\"/>\n","   </g>\n","   <g id=\"matplotlib.axis_1\">\n","    <g id=\"xtick_1\">\n","     <g id=\"line2d_1\">\n","      <path d=\"M 82.214967 143.1 \n","L 82.214967 7.2 \n","\" clip-path=\"url(#paf1508e7f6)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n","     </g>\n","     <g id=\"line2d_2\">\n","      <defs>\n","       <path id=\"m3a97404132\" d=\"M 0 0 \n","L 0 3.5 \n","\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n","      </defs>\n","      <g>\n","       <use xlink:href=\"#m3a97404132\" x=\"82.214967\" y=\"143.1\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n","      </g>\n","     </g>\n","     <g id=\"text_1\">\n","      <!-- 20 -->\n","      <g transform=\"translate(75.852467 157.698438)scale(0.1 -0.1)\">\n","       <defs>\n","        <path id=\"DejaVuSans-32\" d=\"M 1228 531 \n","L 3431 531 \n","L 3431 0 \n","L 469 0 \n","L 469 531 \n","Q 828 903 1448 1529 \n","Q 2069 2156 2228 2338 \n","Q 2531 2678 2651 2914 \n","Q 2772 3150 2772 3378 \n","Q 2772 3750 2511 3984 \n","Q 2250 4219 1831 4219 \n","Q 1534 4219 1204 4116 \n","Q 875 4013 500 3803 \n","L 500 4441 \n","Q 881 4594 1212 4672 \n","Q 1544 4750 1819 4750 \n","Q 2544 4750 2975 4387 \n","Q 3406 4025 3406 3419 \n","Q 3406 3131 3298 2873 \n","Q 3191 2616 2906 2266 \n","Q 2828 2175 2409 1742 \n","Q 1991 1309 1228 531 \n","z\n","\" transform=\"scale(0.015625)\"/>\n","        <path id=\"DejaVuSans-30\" d=\"M 2034 4250 \n","Q 1547 4250 1301 3770 \n","Q 1056 3291 1056 2328 \n","Q 1056 1369 1301 889 \n","Q 1547 409 2034 409 \n","Q 2525 409 2770 889 \n","Q 3016 1369 3016 2328 \n","Q 3016 3291 2770 3770 \n","Q 2525 4250 2034 4250 \n","z\n","M 2034 4750 \n","Q 2819 4750 3233 4129 \n","Q 3647 3509 3647 2328 \n","Q 3647 1150 3233 529 \n","Q 2819 -91 2034 -91 \n","Q 1250 -91 836 529 \n","Q 422 1150 422 2328 \n","Q 422 3509 836 4129 \n","Q 1250 4750 2034 4750 \n","z\n","\" transform=\"scale(0.015625)\"/>\n","       </defs>\n","       <use xlink:href=\"#DejaVuSans-32\"/>\n","       <use xlink:href=\"#DejaVuSans-30\" x=\"63.623047\"/>\n","      </g>\n","     </g>\n","    </g>\n","    <g id=\"xtick_2\">\n","     <g id=\"line2d_3\">\n","      <path d=\"M 123.330757 143.1 \n","L 123.330757 7.2 \n","\" clip-path=\"url(#paf1508e7f6)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n","     </g>\n","     <g id=\"line2d_4\">\n","      <g>\n","       <use xlink:href=\"#m3a97404132\" x=\"123.330757\" y=\"143.1\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n","      </g>\n","     </g>\n","     <g id=\"text_2\">\n","      <!-- 40 -->\n","      <g transform=\"translate(116.968257 157.698438)scale(0.1 -0.1)\">\n","       <defs>\n","        <path id=\"DejaVuSans-34\" d=\"M 2419 4116 \n","L 825 1625 \n","L 2419 1625 \n","L 2419 4116 \n","z\n","M 2253 4666 \n","L 3047 4666 \n","L 3047 1625 \n","L 3713 1625 \n","L 3713 1100 \n","L 3047 1100 \n","L 3047 0 \n","L 2419 0 \n","L 2419 1100 \n","L 313 1100 \n","L 313 1709 \n","L 2253 4666 \n","z\n","\" transform=\"scale(0.015625)\"/>\n","       </defs>\n","       <use xlink:href=\"#DejaVuSans-34\"/>\n","       <use xlink:href=\"#DejaVuSans-30\" x=\"63.623047\"/>\n","      </g>\n","     </g>\n","    </g>\n","    <g id=\"xtick_3\">\n","     <g id=\"line2d_5\">\n","      <path d=\"M 164.446546 143.1 \n","L 164.446546 7.2 \n","\" clip-path=\"url(#paf1508e7f6)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n","     </g>\n","     <g id=\"line2d_6\">\n","      <g>\n","       <use xlink:href=\"#m3a97404132\" x=\"164.446546\" y=\"143.1\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n","      </g>\n","     </g>\n","     <g id=\"text_3\">\n","      <!-- 60 -->\n","      <g transform=\"translate(158.084046 157.698438)scale(0.1 -0.1)\">\n","       <defs>\n","        <path id=\"DejaVuSans-36\" d=\"M 2113 2584 \n","Q 1688 2584 1439 2293 \n","Q 1191 2003 1191 1497 \n","Q 1191 994 1439 701 \n","Q 1688 409 2113 409 \n","Q 2538 409 2786 701 \n","Q 3034 994 3034 1497 \n","Q 3034 2003 2786 2293 \n","Q 2538 2584 2113 2584 \n","z\n","M 3366 4563 \n","L 3366 3988 \n","Q 3128 4100 2886 4159 \n","Q 2644 4219 2406 4219 \n","Q 1781 4219 1451 3797 \n","Q 1122 3375 1075 2522 \n","Q 1259 2794 1537 2939 \n","Q 1816 3084 2150 3084 \n","Q 2853 3084 3261 2657 \n","Q 3669 2231 3669 1497 \n","Q 3669 778 3244 343 \n","Q 2819 -91 2113 -91 \n","Q 1303 -91 875 529 \n","Q 447 1150 447 2328 \n","Q 447 3434 972 4092 \n","Q 1497 4750 2381 4750 \n","Q 2619 4750 2861 4703 \n","Q 3103 4656 3366 4563 \n","z\n","\" transform=\"scale(0.015625)\"/>\n","       </defs>\n","       <use xlink:href=\"#DejaVuSans-36\"/>\n","       <use xlink:href=\"#DejaVuSans-30\" x=\"63.623047\"/>\n","      </g>\n","     </g>\n","    </g>\n","    <g id=\"xtick_4\">\n","     <g id=\"line2d_7\">\n","      <path d=\"M 205.562336 143.1 \n","L 205.562336 7.2 \n","\" clip-path=\"url(#paf1508e7f6)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n","     </g>\n","     <g id=\"line2d_8\">\n","      <g>\n","       <use xlink:href=\"#m3a97404132\" x=\"205.562336\" y=\"143.1\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n","      </g>\n","     </g>\n","     <g id=\"text_4\">\n","      <!-- 80 -->\n","      <g transform=\"translate(199.199836 157.698438)scale(0.1 -0.1)\">\n","       <defs>\n","        <path id=\"DejaVuSans-38\" d=\"M 2034 2216 \n","Q 1584 2216 1326 1975 \n","Q 1069 1734 1069 1313 \n","Q 1069 891 1326 650 \n","Q 1584 409 2034 409 \n","Q 2484 409 2743 651 \n","Q 3003 894 3003 1313 \n","Q 3003 1734 2745 1975 \n","Q 2488 2216 2034 2216 \n","z\n","M 1403 2484 \n","Q 997 2584 770 2862 \n","Q 544 3141 544 3541 \n","Q 544 4100 942 4425 \n","Q 1341 4750 2034 4750 \n","Q 2731 4750 3128 4425 \n","Q 3525 4100 3525 3541 \n","Q 3525 3141 3298 2862 \n","Q 3072 2584 2669 2484 \n","Q 3125 2378 3379 2068 \n","Q 3634 1759 3634 1313 \n","Q 3634 634 3220 271 \n","Q 2806 -91 2034 -91 \n","Q 1263 -91 848 271 \n","Q 434 634 434 1313 \n","Q 434 1759 690 2068 \n","Q 947 2378 1403 2484 \n","z\n","M 1172 3481 \n","Q 1172 3119 1398 2916 \n","Q 1625 2713 2034 2713 \n","Q 2441 2713 2670 2916 \n","Q 2900 3119 2900 3481 \n","Q 2900 3844 2670 4047 \n","Q 2441 4250 2034 4250 \n","Q 1625 4250 1398 4047 \n","Q 1172 3844 1172 3481 \n","z\n","\" transform=\"scale(0.015625)\"/>\n","       </defs>\n","       <use xlink:href=\"#DejaVuSans-38\"/>\n","       <use xlink:href=\"#DejaVuSans-30\" x=\"63.623047\"/>\n","      </g>\n","     </g>\n","    </g>\n","    <g id=\"xtick_5\">\n","     <g id=\"line2d_9\">\n","      <path d=\"M 246.678125 143.1 \n","L 246.678125 7.2 \n","\" clip-path=\"url(#paf1508e7f6)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n","     </g>\n","     <g id=\"line2d_10\">\n","      <g>\n","       <use xlink:href=\"#m3a97404132\" x=\"246.678125\" y=\"143.1\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n","      </g>\n","     </g>\n","     <g id=\"text_5\">\n","      <!-- 100 -->\n","      <g transform=\"translate(237.134375 157.698438)scale(0.1 -0.1)\">\n","       <defs>\n","        <path id=\"DejaVuSans-31\" d=\"M 794 531 \n","L 1825 531 \n","L 1825 4091 \n","L 703 3866 \n","L 703 4441 \n","L 1819 4666 \n","L 2450 4666 \n","L 2450 531 \n","L 3481 531 \n","L 3481 0 \n","L 794 0 \n","L 794 531 \n","z\n","\" transform=\"scale(0.015625)\"/>\n","       </defs>\n","       <use xlink:href=\"#DejaVuSans-31\"/>\n","       <use xlink:href=\"#DejaVuSans-30\" x=\"63.623047\"/>\n","       <use xlink:href=\"#DejaVuSans-30\" x=\"127.246094\"/>\n","      </g>\n","     </g>\n","    </g>\n","    <g id=\"text_6\">\n","     <!-- epochs -->\n","     <g transform=\"translate(131.195312 171.376563)scale(0.1 -0.1)\">\n","      <defs>\n","       <path id=\"DejaVuSans-65\" d=\"M 3597 1894 \n","L 3597 1613 \n","L 953 1613 \n","Q 991 1019 1311 708 \n","Q 1631 397 2203 397 \n","Q 2534 397 2845 478 \n","Q 3156 559 3463 722 \n","L 3463 178 \n","Q 3153 47 2828 -22 \n","Q 2503 -91 2169 -91 \n","Q 1331 -91 842 396 \n","Q 353 884 353 1716 \n","Q 353 2575 817 3079 \n","Q 1281 3584 2069 3584 \n","Q 2775 3584 3186 3129 \n","Q 3597 2675 3597 1894 \n","z\n","M 3022 2063 \n","Q 3016 2534 2758 2815 \n","Q 2500 3097 2075 3097 \n","Q 1594 3097 1305 2825 \n","Q 1016 2553 972 2059 \n","L 3022 2063 \n","z\n","\" transform=\"scale(0.015625)\"/>\n","       <path id=\"DejaVuSans-70\" d=\"M 1159 525 \n","L 1159 -1331 \n","L 581 -1331 \n","L 581 3500 \n","L 1159 3500 \n","L 1159 2969 \n","Q 1341 3281 1617 3432 \n","Q 1894 3584 2278 3584 \n","Q 2916 3584 3314 3078 \n","Q 3713 2572 3713 1747 \n","Q 3713 922 3314 415 \n","Q 2916 -91 2278 -91 \n","Q 1894 -91 1617 61 \n","Q 1341 213 1159 525 \n","z\n","M 3116 1747 \n","Q 3116 2381 2855 2742 \n","Q 2594 3103 2138 3103 \n","Q 1681 3103 1420 2742 \n","Q 1159 2381 1159 1747 \n","Q 1159 1113 1420 752 \n","Q 1681 391 2138 391 \n","Q 2594 391 2855 752 \n","Q 3116 1113 3116 1747 \n","z\n","\" transform=\"scale(0.015625)\"/>\n","       <path id=\"DejaVuSans-6f\" d=\"M 1959 3097 \n","Q 1497 3097 1228 2736 \n","Q 959 2375 959 1747 \n","Q 959 1119 1226 758 \n","Q 1494 397 1959 397 \n","Q 2419 397 2687 759 \n","Q 2956 1122 2956 1747 \n","Q 2956 2369 2687 2733 \n","Q 2419 3097 1959 3097 \n","z\n","M 1959 3584 \n","Q 2709 3584 3137 3096 \n","Q 3566 2609 3566 1747 \n","Q 3566 888 3137 398 \n","Q 2709 -91 1959 -91 \n","Q 1206 -91 779 398 \n","Q 353 888 353 1747 \n","Q 353 2609 779 3096 \n","Q 1206 3584 1959 3584 \n","z\n","\" transform=\"scale(0.015625)\"/>\n","       <path id=\"DejaVuSans-63\" d=\"M 3122 3366 \n","L 3122 2828 \n","Q 2878 2963 2633 3030 \n","Q 2388 3097 2138 3097 \n","Q 1578 3097 1268 2742 \n","Q 959 2388 959 1747 \n","Q 959 1106 1268 751 \n","Q 1578 397 2138 397 \n","Q 2388 397 2633 464 \n","Q 2878 531 3122 666 \n","L 3122 134 \n","Q 2881 22 2623 -34 \n","Q 2366 -91 2075 -91 \n","Q 1284 -91 818 406 \n","Q 353 903 353 1747 \n","Q 353 2603 823 3093 \n","Q 1294 3584 2113 3584 \n","Q 2378 3584 2631 3529 \n","Q 2884 3475 3122 3366 \n","z\n","\" transform=\"scale(0.015625)\"/>\n","       <path id=\"DejaVuSans-68\" d=\"M 3513 2113 \n","L 3513 0 \n","L 2938 0 \n","L 2938 2094 \n","Q 2938 2591 2744 2837 \n","Q 2550 3084 2163 3084 \n","Q 1697 3084 1428 2787 \n","Q 1159 2491 1159 1978 \n","L 1159 0 \n","L 581 0 \n","L 581 4863 \n","L 1159 4863 \n","L 1159 2956 \n","Q 1366 3272 1645 3428 \n","Q 1925 3584 2291 3584 \n","Q 2894 3584 3203 3211 \n","Q 3513 2838 3513 2113 \n","z\n","\" transform=\"scale(0.015625)\"/>\n","       <path id=\"DejaVuSans-73\" d=\"M 2834 3397 \n","L 2834 2853 \n","Q 2591 2978 2328 3040 \n","Q 2066 3103 1784 3103 \n","Q 1356 3103 1142 2972 \n","Q 928 2841 928 2578 \n","Q 928 2378 1081 2264 \n","Q 1234 2150 1697 2047 \n","L 1894 2003 \n","Q 2506 1872 2764 1633 \n","Q 3022 1394 3022 966 \n","Q 3022 478 2636 193 \n","Q 2250 -91 1575 -91 \n","Q 1294 -91 989 -36 \n","Q 684 19 347 128 \n","L 347 722 \n","Q 666 556 975 473 \n","Q 1284 391 1588 391 \n","Q 1994 391 2212 530 \n","Q 2431 669 2431 922 \n","Q 2431 1156 2273 1281 \n","Q 2116 1406 1581 1522 \n","L 1381 1569 \n","Q 847 1681 609 1914 \n","Q 372 2147 372 2553 \n","Q 372 3047 722 3315 \n","Q 1072 3584 1716 3584 \n","Q 2034 3584 2315 3537 \n","Q 2597 3491 2834 3397 \n","z\n","\" transform=\"scale(0.015625)\"/>\n","      </defs>\n","      <use xlink:href=\"#DejaVuSans-65\"/>\n","      <use xlink:href=\"#DejaVuSans-70\" x=\"61.523438\"/>\n","      <use xlink:href=\"#DejaVuSans-6f\" x=\"125\"/>\n","      <use xlink:href=\"#DejaVuSans-63\" x=\"186.181641\"/>\n","      <use xlink:href=\"#DejaVuSans-68\" x=\"241.162109\"/>\n","      <use xlink:href=\"#DejaVuSans-73\" x=\"304.541016\"/>\n","     </g>\n","    </g>\n","   </g>\n","   <g id=\"matplotlib.axis_2\">\n","    <g id=\"ytick_1\">\n","     <g id=\"line2d_11\">\n","      <path d=\"M 51.378125 123.123527 \n","L 246.678125 123.123527 \n","\" clip-path=\"url(#paf1508e7f6)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n","     </g>\n","     <g id=\"line2d_12\">\n","      <defs>\n","       <path id=\"m35267bf620\" d=\"M 0 0 \n","L -3.5 0 \n","\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n","      </defs>\n","      <g>\n","       <use xlink:href=\"#m35267bf620\" x=\"51.378125\" y=\"123.123527\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n","      </g>\n","     </g>\n","     <g id=\"text_7\">\n","      <!-- $\\mathdefault{10^{-5}}$ -->\n","      <g transform=\"translate(20.878125 126.922746)scale(0.1 -0.1)\">\n","       <defs>\n","        <path id=\"DejaVuSans-2212\" d=\"M 678 2272 \n","L 4684 2272 \n","L 4684 1741 \n","L 678 1741 \n","L 678 2272 \n","z\n","\" transform=\"scale(0.015625)\"/>\n","        <path id=\"DejaVuSans-35\" d=\"M 691 4666 \n","L 3169 4666 \n","L 3169 4134 \n","L 1269 4134 \n","L 1269 2991 \n","Q 1406 3038 1543 3061 \n","Q 1681 3084 1819 3084 \n","Q 2600 3084 3056 2656 \n","Q 3513 2228 3513 1497 \n","Q 3513 744 3044 326 \n","Q 2575 -91 1722 -91 \n","Q 1428 -91 1123 -41 \n","Q 819 9 494 109 \n","L 494 744 \n","Q 775 591 1075 516 \n","Q 1375 441 1709 441 \n","Q 2250 441 2565 725 \n","Q 2881 1009 2881 1497 \n","Q 2881 1984 2565 2268 \n","Q 2250 2553 1709 2553 \n","Q 1456 2553 1204 2497 \n","Q 953 2441 691 2322 \n","L 691 4666 \n","z\n","\" transform=\"scale(0.015625)\"/>\n","       </defs>\n","       <use xlink:href=\"#DejaVuSans-31\" transform=\"translate(0 0.684375)\"/>\n","       <use xlink:href=\"#DejaVuSans-30\" transform=\"translate(63.623047 0.684375)\"/>\n","       <use xlink:href=\"#DejaVuSans-2212\" transform=\"translate(128.203125 38.965625)scale(0.7)\"/>\n","       <use xlink:href=\"#DejaVuSans-35\" transform=\"translate(186.855469 38.965625)scale(0.7)\"/>\n","      </g>\n","     </g>\n","    </g>\n","    <g id=\"ytick_2\">\n","     <g id=\"line2d_13\">\n","      <path d=\"M 51.378125 91.281762 \n","L 246.678125 91.281762 \n","\" clip-path=\"url(#paf1508e7f6)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n","     </g>\n","     <g id=\"line2d_14\">\n","      <g>\n","       <use xlink:href=\"#m35267bf620\" x=\"51.378125\" y=\"91.281762\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n","      </g>\n","     </g>\n","     <g id=\"text_8\">\n","      <!-- $\\mathdefault{10^{-3}}$ -->\n","      <g transform=\"translate(20.878125 95.080981)scale(0.1 -0.1)\">\n","       <defs>\n","        <path id=\"DejaVuSans-33\" d=\"M 2597 2516 \n","Q 3050 2419 3304 2112 \n","Q 3559 1806 3559 1356 \n","Q 3559 666 3084 287 \n","Q 2609 -91 1734 -91 \n","Q 1441 -91 1130 -33 \n","Q 819 25 488 141 \n","L 488 750 \n","Q 750 597 1062 519 \n","Q 1375 441 1716 441 \n","Q 2309 441 2620 675 \n","Q 2931 909 2931 1356 \n","Q 2931 1769 2642 2001 \n","Q 2353 2234 1838 2234 \n","L 1294 2234 \n","L 1294 2753 \n","L 1863 2753 \n","Q 2328 2753 2575 2939 \n","Q 2822 3125 2822 3475 \n","Q 2822 3834 2567 4026 \n","Q 2313 4219 1838 4219 \n","Q 1578 4219 1281 4162 \n","Q 984 4106 628 3988 \n","L 628 4550 \n","Q 988 4650 1302 4700 \n","Q 1616 4750 1894 4750 \n","Q 2613 4750 3031 4423 \n","Q 3450 4097 3450 3541 \n","Q 3450 3153 3228 2886 \n","Q 3006 2619 2597 2516 \n","z\n","\" transform=\"scale(0.015625)\"/>\n","       </defs>\n","       <use xlink:href=\"#DejaVuSans-31\" transform=\"translate(0 0.765625)\"/>\n","       <use xlink:href=\"#DejaVuSans-30\" transform=\"translate(63.623047 0.765625)\"/>\n","       <use xlink:href=\"#DejaVuSans-2212\" transform=\"translate(128.203125 39.046875)scale(0.7)\"/>\n","       <use xlink:href=\"#DejaVuSans-33\" transform=\"translate(186.855469 39.046875)scale(0.7)\"/>\n","      </g>\n","     </g>\n","    </g>\n","    <g id=\"ytick_3\">\n","     <g id=\"line2d_15\">\n","      <path d=\"M 51.378125 59.439996 \n","L 246.678125 59.439996 \n","\" clip-path=\"url(#paf1508e7f6)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n","     </g>\n","     <g id=\"line2d_16\">\n","      <g>\n","       <use xlink:href=\"#m35267bf620\" x=\"51.378125\" y=\"59.439996\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n","      </g>\n","     </g>\n","     <g id=\"text_9\">\n","      <!-- $\\mathdefault{10^{-1}}$ -->\n","      <g transform=\"translate(20.878125 63.239215)scale(0.1 -0.1)\">\n","       <use xlink:href=\"#DejaVuSans-31\" transform=\"translate(0 0.684375)\"/>\n","       <use xlink:href=\"#DejaVuSans-30\" transform=\"translate(63.623047 0.684375)\"/>\n","       <use xlink:href=\"#DejaVuSans-2212\" transform=\"translate(128.203125 38.965625)scale(0.7)\"/>\n","       <use xlink:href=\"#DejaVuSans-31\" transform=\"translate(186.855469 38.965625)scale(0.7)\"/>\n","      </g>\n","     </g>\n","    </g>\n","    <g id=\"ytick_4\">\n","     <g id=\"line2d_17\">\n","      <path d=\"M 51.378125 27.598231 \n","L 246.678125 27.598231 \n","\" clip-path=\"url(#paf1508e7f6)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n","     </g>\n","     <g id=\"line2d_18\">\n","      <g>\n","       <use xlink:href=\"#m35267bf620\" x=\"51.378125\" y=\"27.598231\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n","      </g>\n","     </g>\n","     <g id=\"text_10\">\n","      <!-- $\\mathdefault{10^{1}}$ -->\n","      <g transform=\"translate(26.778125 31.397449)scale(0.1 -0.1)\">\n","       <use xlink:href=\"#DejaVuSans-31\" transform=\"translate(0 0.684375)\"/>\n","       <use xlink:href=\"#DejaVuSans-30\" transform=\"translate(63.623047 0.684375)\"/>\n","       <use xlink:href=\"#DejaVuSans-31\" transform=\"translate(128.203125 38.965625)scale(0.7)\"/>\n","      </g>\n","     </g>\n","    </g>\n","    <g id=\"text_11\">\n","     <!-- loss -->\n","     <g transform=\"translate(14.798437 84.807812)rotate(-90)scale(0.1 -0.1)\">\n","      <defs>\n","       <path id=\"DejaVuSans-6c\" d=\"M 603 4863 \n","L 1178 4863 \n","L 1178 0 \n","L 603 0 \n","L 603 4863 \n","z\n","\" transform=\"scale(0.015625)\"/>\n","      </defs>\n","      <use xlink:href=\"#DejaVuSans-6c\"/>\n","      <use xlink:href=\"#DejaVuSans-6f\" x=\"27.783203\"/>\n","      <use xlink:href=\"#DejaVuSans-73\" x=\"88.964844\"/>\n","      <use xlink:href=\"#DejaVuSans-73\" x=\"141.064453\"/>\n","     </g>\n","    </g>\n","   </g>\n","   <g id=\"line2d_19\">\n","    <path d=\"M 51.378125 26.106104 \n","L 61.657072 35.21742 \n","L 71.93602 43.468449 \n","L 82.214967 51.117298 \n","L 92.493914 58.352648 \n","L 102.772862 65.218746 \n","L 113.051809 71.755366 \n","L 123.330757 77.981748 \n","L 133.609704 83.887567 \n","L 143.888651 89.524081 \n","L 154.167599 94.890499 \n","L 164.446546 100.052997 \n","L 174.725493 105.018093 \n","L 185.004441 109.832648 \n","L 195.283388 114.5259 \n","L 205.562336 119.116875 \n","L 215.841283 123.634023 \n","L 226.12023 128.098305 \n","L 236.399178 132.53015 \n","L 246.678125 136.922727 \n","\" clip-path=\"url(#paf1508e7f6)\" style=\"fill: none; stroke: #1f77b4; stroke-width: 1.5; stroke-linecap: square\"/>\n","   </g>\n","   <g id=\"line2d_20\">\n","    <path d=\"M 51.378125 13.38739 \n","L 61.657072 13.41182 \n","L 71.93602 13.407508 \n","L 82.214967 13.399341 \n","L 92.493914 13.392222 \n","L 102.772862 13.386754 \n","L 113.051809 13.383239 \n","L 123.330757 13.380876 \n","L 133.609704 13.379493 \n","L 143.888651 13.378573 \n","L 154.167599 13.378049 \n","L 164.446546 13.377708 \n","L 174.725493 13.377516 \n","L 185.004441 13.377404 \n","L 195.283388 13.377345 \n","L 205.562336 13.377309 \n","L 215.841283 13.377291 \n","L 226.12023 13.377279 \n","L 236.399178 13.377276 \n","L 246.678125 13.377273 \n","\" clip-path=\"url(#paf1508e7f6)\" style=\"fill: none; stroke-dasharray: 5.55,2.4; stroke-dashoffset: 0; stroke: #bf00bf; stroke-width: 1.5\"/>\n","   </g>\n","   <g id=\"patch_3\">\n","    <path d=\"M 51.378125 143.1 \n","L 51.378125 7.2 \n","\" style=\"fill: none; stroke: #000000; stroke-width: 0.8; stroke-linejoin: miter; stroke-linecap: square\"/>\n","   </g>\n","   <g id=\"patch_4\">\n","    <path d=\"M 246.678125 143.1 \n","L 246.678125 7.2 \n","\" style=\"fill: none; stroke: #000000; stroke-width: 0.8; stroke-linejoin: miter; stroke-linecap: square\"/>\n","   </g>\n","   <g id=\"patch_5\">\n","    <path d=\"M 51.378125 143.1 \n","L 246.678125 143.1 \n","\" style=\"fill: none; stroke: #000000; stroke-width: 0.8; stroke-linejoin: miter; stroke-linecap: square\"/>\n","   </g>\n","   <g id=\"patch_6\">\n","    <path d=\"M 51.378125 7.2 \n","L 246.678125 7.2 \n","\" style=\"fill: none; stroke: #000000; stroke-width: 0.8; stroke-linejoin: miter; stroke-linecap: square\"/>\n","   </g>\n","   <g id=\"legend_1\">\n","    <g id=\"patch_7\">\n","     <path d=\"M 58.378125 138.1 \n","L 113.653125 138.1 \n","Q 115.653125 138.1 115.653125 136.1 \n","L 115.653125 107.74375 \n","Q 115.653125 105.74375 113.653125 105.74375 \n","L 58.378125 105.74375 \n","Q 56.378125 105.74375 56.378125 107.74375 \n","L 56.378125 136.1 \n","Q 56.378125 138.1 58.378125 138.1 \n","z\n","\" style=\"fill: #ffffff; opacity: 0.8; stroke: #cccccc; stroke-linejoin: miter\"/>\n","    </g>\n","    <g id=\"line2d_21\">\n","     <path d=\"M 60.378125 113.842187 \n","L 70.378125 113.842187 \n","L 80.378125 113.842187 \n","\" style=\"fill: none; stroke: #1f77b4; stroke-width: 1.5; stroke-linecap: square\"/>\n","    </g>\n","    <g id=\"text_12\">\n","     <!-- train -->\n","     <g transform=\"translate(88.378125 117.342187)scale(0.1 -0.1)\">\n","      <defs>\n","       <path id=\"DejaVuSans-74\" d=\"M 1172 4494 \n","L 1172 3500 \n","L 2356 3500 \n","L 2356 3053 \n","L 1172 3053 \n","L 1172 1153 \n","Q 1172 725 1289 603 \n","Q 1406 481 1766 481 \n","L 2356 481 \n","L 2356 0 \n","L 1766 0 \n","Q 1100 0 847 248 \n","Q 594 497 594 1153 \n","L 594 3053 \n","L 172 3053 \n","L 172 3500 \n","L 594 3500 \n","L 594 4494 \n","L 1172 4494 \n","z\n","\" transform=\"scale(0.015625)\"/>\n","       <path id=\"DejaVuSans-72\" d=\"M 2631 2963 \n","Q 2534 3019 2420 3045 \n","Q 2306 3072 2169 3072 \n","Q 1681 3072 1420 2755 \n","Q 1159 2438 1159 1844 \n","L 1159 0 \n","L 581 0 \n","L 581 3500 \n","L 1159 3500 \n","L 1159 2956 \n","Q 1341 3275 1631 3429 \n","Q 1922 3584 2338 3584 \n","Q 2397 3584 2469 3576 \n","Q 2541 3569 2628 3553 \n","L 2631 2963 \n","z\n","\" transform=\"scale(0.015625)\"/>\n","       <path id=\"DejaVuSans-61\" d=\"M 2194 1759 \n","Q 1497 1759 1228 1600 \n","Q 959 1441 959 1056 \n","Q 959 750 1161 570 \n","Q 1363 391 1709 391 \n","Q 2188 391 2477 730 \n","Q 2766 1069 2766 1631 \n","L 2766 1759 \n","L 2194 1759 \n","z\n","M 3341 1997 \n","L 3341 0 \n","L 2766 0 \n","L 2766 531 \n","Q 2569 213 2275 61 \n","Q 1981 -91 1556 -91 \n","Q 1019 -91 701 211 \n","Q 384 513 384 1019 \n","Q 384 1609 779 1909 \n","Q 1175 2209 1959 2209 \n","L 2766 2209 \n","L 2766 2266 \n","Q 2766 2663 2505 2880 \n","Q 2244 3097 1772 3097 \n","Q 1472 3097 1187 3025 \n","Q 903 2953 641 2809 \n","L 641 3341 \n","Q 956 3463 1253 3523 \n","Q 1550 3584 1831 3584 \n","Q 2591 3584 2966 3190 \n","Q 3341 2797 3341 1997 \n","z\n","\" transform=\"scale(0.015625)\"/>\n","       <path id=\"DejaVuSans-69\" d=\"M 603 3500 \n","L 1178 3500 \n","L 1178 0 \n","L 603 0 \n","L 603 3500 \n","z\n","M 603 4863 \n","L 1178 4863 \n","L 1178 4134 \n","L 603 4134 \n","L 603 4863 \n","z\n","\" transform=\"scale(0.015625)\"/>\n","       <path id=\"DejaVuSans-6e\" d=\"M 3513 2113 \n","L 3513 0 \n","L 2938 0 \n","L 2938 2094 \n","Q 2938 2591 2744 2837 \n","Q 2550 3084 2163 3084 \n","Q 1697 3084 1428 2787 \n","Q 1159 2491 1159 1978 \n","L 1159 0 \n","L 581 0 \n","L 581 3500 \n","L 1159 3500 \n","L 1159 2956 \n","Q 1366 3272 1645 3428 \n","Q 1925 3584 2291 3584 \n","Q 2894 3584 3203 3211 \n","Q 3513 2838 3513 2113 \n","z\n","\" transform=\"scale(0.015625)\"/>\n","      </defs>\n","      <use xlink:href=\"#DejaVuSans-74\"/>\n","      <use xlink:href=\"#DejaVuSans-72\" x=\"39.208984\"/>\n","      <use xlink:href=\"#DejaVuSans-61\" x=\"80.322266\"/>\n","      <use xlink:href=\"#DejaVuSans-69\" x=\"141.601562\"/>\n","      <use xlink:href=\"#DejaVuSans-6e\" x=\"169.384766\"/>\n","     </g>\n","    </g>\n","    <g id=\"line2d_22\">\n","     <path d=\"M 60.378125 128.520312 \n","L 70.378125 128.520312 \n","L 80.378125 128.520312 \n","\" style=\"fill: none; stroke-dasharray: 5.55,2.4; stroke-dashoffset: 0; stroke: #bf00bf; stroke-width: 1.5\"/>\n","    </g>\n","    <g id=\"text_13\">\n","     <!-- test -->\n","     <g transform=\"translate(88.378125 132.020312)scale(0.1 -0.1)\">\n","      <use xlink:href=\"#DejaVuSans-74\"/>\n","      <use xlink:href=\"#DejaVuSans-65\" x=\"39.208984\"/>\n","      <use xlink:href=\"#DejaVuSans-73\" x=\"100.732422\"/>\n","      <use xlink:href=\"#DejaVuSans-74\" x=\"152.832031\"/>\n","     </g>\n","    </g>\n","   </g>\n","  </g>\n"," </g>\n"," <defs>\n","  <clipPath id=\"paf1508e7f6\">\n","   <rect x=\"51.378125\" y=\"7.2\" width=\"195.3\" height=\"135.9\"/>\n","  </clipPath>\n"," </defs>\n","</svg>\n"],"text/plain":["<Figure size 252x180 with 1 Axes>"]},"metadata":{"needs_background":"light"},"output_type":"display_data"}],"source":["train(lambd=0)"],"id":"045cb210"},{"cell_type":"markdown","metadata":{"origin_pos":20,"id":"108f11e4"},"source":["### [**使用权重衰减**]\n","\n","下面，我们使用权重衰减来运行代码。\n","注意，在这里训练误差增大，但测试误差减小。\n","这正是我们期望从正则化中得到的效果。\n"],"id":"108f11e4"},{"cell_type":"code","execution_count":null,"metadata":{"execution":{"iopub.execute_input":"2022-07-31T02:22:55.835902Z","iopub.status.busy":"2022-07-31T02:22:55.835673Z","iopub.status.idle":"2022-07-31T02:23:01.979648Z","shell.execute_reply":"2022-07-31T02:23:01.978765Z"},"origin_pos":21,"tab":["pytorch"],"id":"87af15b2","outputId":"a9e1657b-9711-4d71-e957-c4ae96b7a725"},"outputs":[{"name":"stdout","output_type":"stream","text":["w的L2范数是： 0.3558707535266876\n"]},{"data":{"image/svg+xml":["<?xml version=\"1.0\" encoding=\"utf-8\" standalone=\"no\"?>\n","<!DOCTYPE svg PUBLIC \"-//W3C//DTD SVG 1.1//EN\"\n","  \"http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd\">\n","<svg xmlns:xlink=\"http://www.w3.org/1999/xlink\" width=\"263.421875pt\" height=\"182.600266pt\" viewBox=\"0 0 263.421875 182.600266\" xmlns=\"http://www.w3.org/2000/svg\" version=\"1.1\">\n"," <metadata>\n","  <rdf:RDF xmlns:dc=\"http://purl.org/dc/elements/1.1/\" xmlns:cc=\"http://creativecommons.org/ns#\" xmlns:rdf=\"http://www.w3.org/1999/02/22-rdf-syntax-ns#\">\n","   <cc:Work>\n","    <dc:type rdf:resource=\"http://purl.org/dc/dcmitype/StillImage\"/>\n","    <dc:date>2022-07-31T02:23:01.916702</dc:date>\n","    <dc:format>image/svg+xml</dc:format>\n","    <dc:creator>\n","     <cc:Agent>\n","      <dc:title>Matplotlib v3.5.1, https://matplotlib.org/</dc:title>\n","     </cc:Agent>\n","    </dc:creator>\n","   </cc:Work>\n","  </rdf:RDF>\n"," </metadata>\n"," <defs>\n","  <style type=\"text/css\">*{stroke-linejoin: round; stroke-linecap: butt}</style>\n"," </defs>\n"," <g id=\"figure_1\">\n","  <g id=\"patch_1\">\n","   <path d=\"M 0 182.600266 \n","L 263.421875 182.600266 \n","L 263.421875 -0 \n","L 0 -0 \n","L 0 182.600266 \n","z\n","\" style=\"fill: none\"/>\n","  </g>\n","  <g id=\"axes_1\">\n","   <g id=\"patch_2\">\n","    <path d=\"M 51.378125 145.044016 \n","L 246.678125 145.044016 \n","L 246.678125 9.144016 \n","L 51.378125 9.144016 \n","z\n","\" style=\"fill: #ffffff\"/>\n","   </g>\n","   <g id=\"matplotlib.axis_1\">\n","    <g id=\"xtick_1\">\n","     <g id=\"line2d_1\">\n","      <path d=\"M 82.214967 145.044016 \n","L 82.214967 9.144016 \n","\" clip-path=\"url(#pd83c5e35c7)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n","     </g>\n","     <g id=\"line2d_2\">\n","      <defs>\n","       <path id=\"mc4b0c12c4b\" d=\"M 0 0 \n","L 0 3.5 \n","\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n","      </defs>\n","      <g>\n","       <use xlink:href=\"#mc4b0c12c4b\" x=\"82.214967\" y=\"145.044016\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n","      </g>\n","     </g>\n","     <g id=\"text_1\">\n","      <!-- 20 -->\n","      <g transform=\"translate(75.852467 159.642454)scale(0.1 -0.1)\">\n","       <defs>\n","        <path id=\"DejaVuSans-32\" d=\"M 1228 531 \n","L 3431 531 \n","L 3431 0 \n","L 469 0 \n","L 469 531 \n","Q 828 903 1448 1529 \n","Q 2069 2156 2228 2338 \n","Q 2531 2678 2651 2914 \n","Q 2772 3150 2772 3378 \n","Q 2772 3750 2511 3984 \n","Q 2250 4219 1831 4219 \n","Q 1534 4219 1204 4116 \n","Q 875 4013 500 3803 \n","L 500 4441 \n","Q 881 4594 1212 4672 \n","Q 1544 4750 1819 4750 \n","Q 2544 4750 2975 4387 \n","Q 3406 4025 3406 3419 \n","Q 3406 3131 3298 2873 \n","Q 3191 2616 2906 2266 \n","Q 2828 2175 2409 1742 \n","Q 1991 1309 1228 531 \n","z\n","\" transform=\"scale(0.015625)\"/>\n","        <path id=\"DejaVuSans-30\" d=\"M 2034 4250 \n","Q 1547 4250 1301 3770 \n","Q 1056 3291 1056 2328 \n","Q 1056 1369 1301 889 \n","Q 1547 409 2034 409 \n","Q 2525 409 2770 889 \n","Q 3016 1369 3016 2328 \n","Q 3016 3291 2770 3770 \n","Q 2525 4250 2034 4250 \n","z\n","M 2034 4750 \n","Q 2819 4750 3233 4129 \n","Q 3647 3509 3647 2328 \n","Q 3647 1150 3233 529 \n","Q 2819 -91 2034 -91 \n","Q 1250 -91 836 529 \n","Q 422 1150 422 2328 \n","Q 422 3509 836 4129 \n","Q 1250 4750 2034 4750 \n","z\n","\" transform=\"scale(0.015625)\"/>\n","       </defs>\n","       <use xlink:href=\"#DejaVuSans-32\"/>\n","       <use xlink:href=\"#DejaVuSans-30\" x=\"63.623047\"/>\n","      </g>\n","     </g>\n","    </g>\n","    <g id=\"xtick_2\">\n","     <g id=\"line2d_3\">\n","      <path d=\"M 123.330757 145.044016 \n","L 123.330757 9.144016 \n","\" clip-path=\"url(#pd83c5e35c7)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n","     </g>\n","     <g id=\"line2d_4\">\n","      <g>\n","       <use xlink:href=\"#mc4b0c12c4b\" x=\"123.330757\" y=\"145.044016\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n","      </g>\n","     </g>\n","     <g id=\"text_2\">\n","      <!-- 40 -->\n","      <g transform=\"translate(116.968257 159.642454)scale(0.1 -0.1)\">\n","       <defs>\n","        <path id=\"DejaVuSans-34\" d=\"M 2419 4116 \n","L 825 1625 \n","L 2419 1625 \n","L 2419 4116 \n","z\n","M 2253 4666 \n","L 3047 4666 \n","L 3047 1625 \n","L 3713 1625 \n","L 3713 1100 \n","L 3047 1100 \n","L 3047 0 \n","L 2419 0 \n","L 2419 1100 \n","L 313 1100 \n","L 313 1709 \n","L 2253 4666 \n","z\n","\" transform=\"scale(0.015625)\"/>\n","       </defs>\n","       <use xlink:href=\"#DejaVuSans-34\"/>\n","       <use xlink:href=\"#DejaVuSans-30\" x=\"63.623047\"/>\n","      </g>\n","     </g>\n","    </g>\n","    <g id=\"xtick_3\">\n","     <g id=\"line2d_5\">\n","      <path d=\"M 164.446546 145.044016 \n","L 164.446546 9.144016 \n","\" clip-path=\"url(#pd83c5e35c7)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n","     </g>\n","     <g id=\"line2d_6\">\n","      <g>\n","       <use xlink:href=\"#mc4b0c12c4b\" x=\"164.446546\" y=\"145.044016\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n","      </g>\n","     </g>\n","     <g id=\"text_3\">\n","      <!-- 60 -->\n","      <g transform=\"translate(158.084046 159.642454)scale(0.1 -0.1)\">\n","       <defs>\n","        <path id=\"DejaVuSans-36\" d=\"M 2113 2584 \n","Q 1688 2584 1439 2293 \n","Q 1191 2003 1191 1497 \n","Q 1191 994 1439 701 \n","Q 1688 409 2113 409 \n","Q 2538 409 2786 701 \n","Q 3034 994 3034 1497 \n","Q 3034 2003 2786 2293 \n","Q 2538 2584 2113 2584 \n","z\n","M 3366 4563 \n","L 3366 3988 \n","Q 3128 4100 2886 4159 \n","Q 2644 4219 2406 4219 \n","Q 1781 4219 1451 3797 \n","Q 1122 3375 1075 2522 \n","Q 1259 2794 1537 2939 \n","Q 1816 3084 2150 3084 \n","Q 2853 3084 3261 2657 \n","Q 3669 2231 3669 1497 \n","Q 3669 778 3244 343 \n","Q 2819 -91 2113 -91 \n","Q 1303 -91 875 529 \n","Q 447 1150 447 2328 \n","Q 447 3434 972 4092 \n","Q 1497 4750 2381 4750 \n","Q 2619 4750 2861 4703 \n","Q 3103 4656 3366 4563 \n","z\n","\" transform=\"scale(0.015625)\"/>\n","       </defs>\n","       <use xlink:href=\"#DejaVuSans-36\"/>\n","       <use xlink:href=\"#DejaVuSans-30\" x=\"63.623047\"/>\n","      </g>\n","     </g>\n","    </g>\n","    <g id=\"xtick_4\">\n","     <g id=\"line2d_7\">\n","      <path d=\"M 205.562336 145.044016 \n","L 205.562336 9.144016 \n","\" clip-path=\"url(#pd83c5e35c7)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n","     </g>\n","     <g id=\"line2d_8\">\n","      <g>\n","       <use xlink:href=\"#mc4b0c12c4b\" x=\"205.562336\" y=\"145.044016\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n","      </g>\n","     </g>\n","     <g id=\"text_4\">\n","      <!-- 80 -->\n","      <g transform=\"translate(199.199836 159.642454)scale(0.1 -0.1)\">\n","       <defs>\n","        <path id=\"DejaVuSans-38\" d=\"M 2034 2216 \n","Q 1584 2216 1326 1975 \n","Q 1069 1734 1069 1313 \n","Q 1069 891 1326 650 \n","Q 1584 409 2034 409 \n","Q 2484 409 2743 651 \n","Q 3003 894 3003 1313 \n","Q 3003 1734 2745 1975 \n","Q 2488 2216 2034 2216 \n","z\n","M 1403 2484 \n","Q 997 2584 770 2862 \n","Q 544 3141 544 3541 \n","Q 544 4100 942 4425 \n","Q 1341 4750 2034 4750 \n","Q 2731 4750 3128 4425 \n","Q 3525 4100 3525 3541 \n","Q 3525 3141 3298 2862 \n","Q 3072 2584 2669 2484 \n","Q 3125 2378 3379 2068 \n","Q 3634 1759 3634 1313 \n","Q 3634 634 3220 271 \n","Q 2806 -91 2034 -91 \n","Q 1263 -91 848 271 \n","Q 434 634 434 1313 \n","Q 434 1759 690 2068 \n","Q 947 2378 1403 2484 \n","z\n","M 1172 3481 \n","Q 1172 3119 1398 2916 \n","Q 1625 2713 2034 2713 \n","Q 2441 2713 2670 2916 \n","Q 2900 3119 2900 3481 \n","Q 2900 3844 2670 4047 \n","Q 2441 4250 2034 4250 \n","Q 1625 4250 1398 4047 \n","Q 1172 3844 1172 3481 \n","z\n","\" transform=\"scale(0.015625)\"/>\n","       </defs>\n","       <use xlink:href=\"#DejaVuSans-38\"/>\n","       <use xlink:href=\"#DejaVuSans-30\" x=\"63.623047\"/>\n","      </g>\n","     </g>\n","    </g>\n","    <g id=\"xtick_5\">\n","     <g id=\"line2d_9\">\n","      <path d=\"M 246.678125 145.044016 \n","L 246.678125 9.144016 \n","\" clip-path=\"url(#pd83c5e35c7)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n","     </g>\n","     <g id=\"line2d_10\">\n","      <g>\n","       <use xlink:href=\"#mc4b0c12c4b\" x=\"246.678125\" y=\"145.044016\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n","      </g>\n","     </g>\n","     <g id=\"text_5\">\n","      <!-- 100 -->\n","      <g transform=\"translate(237.134375 159.642454)scale(0.1 -0.1)\">\n","       <defs>\n","        <path id=\"DejaVuSans-31\" d=\"M 794 531 \n","L 1825 531 \n","L 1825 4091 \n","L 703 3866 \n","L 703 4441 \n","L 1819 4666 \n","L 2450 4666 \n","L 2450 531 \n","L 3481 531 \n","L 3481 0 \n","L 794 0 \n","L 794 531 \n","z\n","\" transform=\"scale(0.015625)\"/>\n","       </defs>\n","       <use xlink:href=\"#DejaVuSans-31\"/>\n","       <use xlink:href=\"#DejaVuSans-30\" x=\"63.623047\"/>\n","       <use xlink:href=\"#DejaVuSans-30\" x=\"127.246094\"/>\n","      </g>\n","     </g>\n","    </g>\n","    <g id=\"text_6\">\n","     <!-- epochs -->\n","     <g transform=\"translate(131.195312 173.320579)scale(0.1 -0.1)\">\n","      <defs>\n","       <path id=\"DejaVuSans-65\" d=\"M 3597 1894 \n","L 3597 1613 \n","L 953 1613 \n","Q 991 1019 1311 708 \n","Q 1631 397 2203 397 \n","Q 2534 397 2845 478 \n","Q 3156 559 3463 722 \n","L 3463 178 \n","Q 3153 47 2828 -22 \n","Q 2503 -91 2169 -91 \n","Q 1331 -91 842 396 \n","Q 353 884 353 1716 \n","Q 353 2575 817 3079 \n","Q 1281 3584 2069 3584 \n","Q 2775 3584 3186 3129 \n","Q 3597 2675 3597 1894 \n","z\n","M 3022 2063 \n","Q 3016 2534 2758 2815 \n","Q 2500 3097 2075 3097 \n","Q 1594 3097 1305 2825 \n","Q 1016 2553 972 2059 \n","L 3022 2063 \n","z\n","\" transform=\"scale(0.015625)\"/>\n","       <path id=\"DejaVuSans-70\" d=\"M 1159 525 \n","L 1159 -1331 \n","L 581 -1331 \n","L 581 3500 \n","L 1159 3500 \n","L 1159 2969 \n","Q 1341 3281 1617 3432 \n","Q 1894 3584 2278 3584 \n","Q 2916 3584 3314 3078 \n","Q 3713 2572 3713 1747 \n","Q 3713 922 3314 415 \n","Q 2916 -91 2278 -91 \n","Q 1894 -91 1617 61 \n","Q 1341 213 1159 525 \n","z\n","M 3116 1747 \n","Q 3116 2381 2855 2742 \n","Q 2594 3103 2138 3103 \n","Q 1681 3103 1420 2742 \n","Q 1159 2381 1159 1747 \n","Q 1159 1113 1420 752 \n","Q 1681 391 2138 391 \n","Q 2594 391 2855 752 \n","Q 3116 1113 3116 1747 \n","z\n","\" transform=\"scale(0.015625)\"/>\n","       <path id=\"DejaVuSans-6f\" d=\"M 1959 3097 \n","Q 1497 3097 1228 2736 \n","Q 959 2375 959 1747 \n","Q 959 1119 1226 758 \n","Q 1494 397 1959 397 \n","Q 2419 397 2687 759 \n","Q 2956 1122 2956 1747 \n","Q 2956 2369 2687 2733 \n","Q 2419 3097 1959 3097 \n","z\n","M 1959 3584 \n","Q 2709 3584 3137 3096 \n","Q 3566 2609 3566 1747 \n","Q 3566 888 3137 398 \n","Q 2709 -91 1959 -91 \n","Q 1206 -91 779 398 \n","Q 353 888 353 1747 \n","Q 353 2609 779 3096 \n","Q 1206 3584 1959 3584 \n","z\n","\" transform=\"scale(0.015625)\"/>\n","       <path id=\"DejaVuSans-63\" d=\"M 3122 3366 \n","L 3122 2828 \n","Q 2878 2963 2633 3030 \n","Q 2388 3097 2138 3097 \n","Q 1578 3097 1268 2742 \n","Q 959 2388 959 1747 \n","Q 959 1106 1268 751 \n","Q 1578 397 2138 397 \n","Q 2388 397 2633 464 \n","Q 2878 531 3122 666 \n","L 3122 134 \n","Q 2881 22 2623 -34 \n","Q 2366 -91 2075 -91 \n","Q 1284 -91 818 406 \n","Q 353 903 353 1747 \n","Q 353 2603 823 3093 \n","Q 1294 3584 2113 3584 \n","Q 2378 3584 2631 3529 \n","Q 2884 3475 3122 3366 \n","z\n","\" transform=\"scale(0.015625)\"/>\n","       <path id=\"DejaVuSans-68\" d=\"M 3513 2113 \n","L 3513 0 \n","L 2938 0 \n","L 2938 2094 \n","Q 2938 2591 2744 2837 \n","Q 2550 3084 2163 3084 \n","Q 1697 3084 1428 2787 \n","Q 1159 2491 1159 1978 \n","L 1159 0 \n","L 581 0 \n","L 581 4863 \n","L 1159 4863 \n","L 1159 2956 \n","Q 1366 3272 1645 3428 \n","Q 1925 3584 2291 3584 \n","Q 2894 3584 3203 3211 \n","Q 3513 2838 3513 2113 \n","z\n","\" transform=\"scale(0.015625)\"/>\n","       <path id=\"DejaVuSans-73\" d=\"M 2834 3397 \n","L 2834 2853 \n","Q 2591 2978 2328 3040 \n","Q 2066 3103 1784 3103 \n","Q 1356 3103 1142 2972 \n","Q 928 2841 928 2578 \n","Q 928 2378 1081 2264 \n","Q 1234 2150 1697 2047 \n","L 1894 2003 \n","Q 2506 1872 2764 1633 \n","Q 3022 1394 3022 966 \n","Q 3022 478 2636 193 \n","Q 2250 -91 1575 -91 \n","Q 1294 -91 989 -36 \n","Q 684 19 347 128 \n","L 347 722 \n","Q 666 556 975 473 \n","Q 1284 391 1588 391 \n","Q 1994 391 2212 530 \n","Q 2431 669 2431 922 \n","Q 2431 1156 2273 1281 \n","Q 2116 1406 1581 1522 \n","L 1381 1569 \n","Q 847 1681 609 1914 \n","Q 372 2147 372 2553 \n","Q 372 3047 722 3315 \n","Q 1072 3584 1716 3584 \n","Q 2034 3584 2315 3537 \n","Q 2597 3491 2834 3397 \n","z\n","\" transform=\"scale(0.015625)\"/>\n","      </defs>\n","      <use xlink:href=\"#DejaVuSans-65\"/>\n","      <use xlink:href=\"#DejaVuSans-70\" x=\"61.523438\"/>\n","      <use xlink:href=\"#DejaVuSans-6f\" x=\"125\"/>\n","      <use xlink:href=\"#DejaVuSans-63\" x=\"186.181641\"/>\n","      <use xlink:href=\"#DejaVuSans-68\" x=\"241.162109\"/>\n","      <use xlink:href=\"#DejaVuSans-73\" x=\"304.541016\"/>\n","     </g>\n","    </g>\n","   </g>\n","   <g id=\"matplotlib.axis_2\">\n","    <g id=\"ytick_1\">\n","     <g id=\"line2d_11\">\n","      <path d=\"M 51.378125 112.581014 \n","L 246.678125 112.581014 \n","\" clip-path=\"url(#pd83c5e35c7)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n","     </g>\n","     <g id=\"line2d_12\">\n","      <defs>\n","       <path id=\"ma8aaa4f60f\" d=\"M 0 0 \n","L -3.5 0 \n","\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n","      </defs>\n","      <g>\n","       <use xlink:href=\"#ma8aaa4f60f\" x=\"51.378125\" y=\"112.581014\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n","      </g>\n","     </g>\n","     <g id=\"text_7\">\n","      <!-- $\\mathdefault{10^{-2}}$ -->\n","      <g transform=\"translate(20.878125 116.380233)scale(0.1 -0.1)\">\n","       <defs>\n","        <path id=\"DejaVuSans-2212\" d=\"M 678 2272 \n","L 4684 2272 \n","L 4684 1741 \n","L 678 1741 \n","L 678 2272 \n","z\n","\" transform=\"scale(0.015625)\"/>\n","       </defs>\n","       <use xlink:href=\"#DejaVuSans-31\" transform=\"translate(0 0.765625)\"/>\n","       <use xlink:href=\"#DejaVuSans-30\" transform=\"translate(63.623047 0.765625)\"/>\n","       <use xlink:href=\"#DejaVuSans-2212\" transform=\"translate(128.203125 39.046875)scale(0.7)\"/>\n","       <use xlink:href=\"#DejaVuSans-32\" transform=\"translate(186.855469 39.046875)scale(0.7)\"/>\n","      </g>\n","     </g>\n","    </g>\n","    <g id=\"ytick_2\">\n","     <g id=\"line2d_13\">\n","      <path d=\"M 51.378125 61.790117 \n","L 246.678125 61.790117 \n","\" clip-path=\"url(#pd83c5e35c7)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n","     </g>\n","     <g id=\"line2d_14\">\n","      <g>\n","       <use xlink:href=\"#ma8aaa4f60f\" x=\"51.378125\" y=\"61.790117\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n","      </g>\n","     </g>\n","     <g id=\"text_8\">\n","      <!-- $\\mathdefault{10^{0}}$ -->\n","      <g transform=\"translate(26.778125 65.589335)scale(0.1 -0.1)\">\n","       <use xlink:href=\"#DejaVuSans-31\" transform=\"translate(0 0.765625)\"/>\n","       <use xlink:href=\"#DejaVuSans-30\" transform=\"translate(63.623047 0.765625)\"/>\n","       <use xlink:href=\"#DejaVuSans-30\" transform=\"translate(128.203125 39.046875)scale(0.7)\"/>\n","      </g>\n","     </g>\n","    </g>\n","    <g id=\"ytick_3\">\n","     <g id=\"line2d_15\">\n","      <path d=\"M 51.378125 10.999219 \n","L 246.678125 10.999219 \n","\" clip-path=\"url(#pd83c5e35c7)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n","     </g>\n","     <g id=\"line2d_16\">\n","      <g>\n","       <use xlink:href=\"#ma8aaa4f60f\" x=\"51.378125\" y=\"10.999219\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n","      </g>\n","     </g>\n","     <g id=\"text_9\">\n","      <!-- $\\mathdefault{10^{2}}$ -->\n","      <g transform=\"translate(26.778125 14.798437)scale(0.1 -0.1)\">\n","       <use xlink:href=\"#DejaVuSans-31\" transform=\"translate(0 0.765625)\"/>\n","       <use xlink:href=\"#DejaVuSans-30\" transform=\"translate(63.623047 0.765625)\"/>\n","       <use xlink:href=\"#DejaVuSans-32\" transform=\"translate(128.203125 39.046875)scale(0.7)\"/>\n","      </g>\n","     </g>\n","    </g>\n","    <g id=\"text_10\">\n","     <!-- loss -->\n","     <g transform=\"translate(14.798437 86.751829)rotate(-90)scale(0.1 -0.1)\">\n","      <defs>\n","       <path id=\"DejaVuSans-6c\" d=\"M 603 4863 \n","L 1178 4863 \n","L 1178 0 \n","L 603 0 \n","L 603 4863 \n","z\n","\" transform=\"scale(0.015625)\"/>\n","      </defs>\n","      <use xlink:href=\"#DejaVuSans-6c\"/>\n","      <use xlink:href=\"#DejaVuSans-6f\" x=\"27.783203\"/>\n","      <use xlink:href=\"#DejaVuSans-73\" x=\"88.964844\"/>\n","      <use xlink:href=\"#DejaVuSans-73\" x=\"141.064453\"/>\n","     </g>\n","    </g>\n","   </g>\n","   <g id=\"line2d_17\">\n","    <path d=\"M 51.378125 34.334134 \n","L 61.657072 50.369663 \n","L 71.93602 65.62548 \n","L 82.214967 80.348757 \n","L 92.493914 94.697254 \n","L 102.772862 108.615806 \n","L 113.051809 121.476602 \n","L 123.330757 131.489245 \n","L 133.609704 136.728038 \n","L 143.888651 138.225215 \n","L 154.167599 138.387269 \n","L 164.446546 138.194005 \n","L 174.725493 138.189174 \n","L 185.004441 138.179419 \n","L 195.283388 138.096809 \n","L 205.562336 138.364611 \n","L 215.841283 138.475723 \n","L 226.12023 138.682048 \n","L 236.399178 138.846207 \n","L 246.678125 138.866744 \n","\" clip-path=\"url(#pd83c5e35c7)\" style=\"fill: none; stroke: #1f77b4; stroke-width: 1.5; stroke-linecap: square\"/>\n","   </g>\n","   <g id=\"line2d_18\">\n","    <path d=\"M 51.378125 15.321289 \n","L 61.657072 19.257826 \n","L 71.93602 23.181495 \n","L 82.214967 27.1216 \n","L 92.493914 31.078954 \n","L 102.772862 35.050837 \n","L 113.051809 39.032385 \n","L 123.330757 43.02071 \n","L 133.609704 47.012623 \n","L 143.888651 51.005697 \n","L 154.167599 54.99629 \n","L 164.446546 58.980406 \n","L 174.725493 62.952397 \n","L 185.004441 66.90531 \n","L 195.283388 70.827173 \n","L 205.562336 74.703102 \n","L 215.841283 78.512446 \n","L 226.12023 82.224176 \n","L 236.399178 85.803689 \n","L 246.678125 89.204236 \n","\" clip-path=\"url(#pd83c5e35c7)\" style=\"fill: none; stroke-dasharray: 5.55,2.4; stroke-dashoffset: 0; stroke: #bf00bf; stroke-width: 1.5\"/>\n","   </g>\n","   <g id=\"patch_3\">\n","    <path d=\"M 51.378125 145.044016 \n","L 51.378125 9.144016 \n","\" style=\"fill: none; stroke: #000000; stroke-width: 0.8; stroke-linejoin: miter; stroke-linecap: square\"/>\n","   </g>\n","   <g id=\"patch_4\">\n","    <path d=\"M 246.678125 145.044016 \n","L 246.678125 9.144016 \n","\" style=\"fill: none; stroke: #000000; stroke-width: 0.8; stroke-linejoin: miter; stroke-linecap: square\"/>\n","   </g>\n","   <g id=\"patch_5\">\n","    <path d=\"M 51.378125 145.044016 \n","L 246.678125 145.044016 \n","\" style=\"fill: none; stroke: #000000; stroke-width: 0.8; stroke-linejoin: miter; stroke-linecap: square\"/>\n","   </g>\n","   <g id=\"patch_6\">\n","    <path d=\"M 51.378125 9.144016 \n","L 246.678125 9.144016 \n","\" style=\"fill: none; stroke: #000000; stroke-width: 0.8; stroke-linejoin: miter; stroke-linecap: square\"/>\n","   </g>\n","   <g id=\"legend_1\">\n","    <g id=\"patch_7\">\n","     <path d=\"M 184.403125 46.500266 \n","L 239.678125 46.500266 \n","Q 241.678125 46.500266 241.678125 44.500266 \n","L 241.678125 16.144016 \n","Q 241.678125 14.144016 239.678125 14.144016 \n","L 184.403125 14.144016 \n","Q 182.403125 14.144016 182.403125 16.144016 \n","L 182.403125 44.500266 \n","Q 182.403125 46.500266 184.403125 46.500266 \n","z\n","\" style=\"fill: #ffffff; opacity: 0.8; stroke: #cccccc; stroke-linejoin: miter\"/>\n","    </g>\n","    <g id=\"line2d_19\">\n","     <path d=\"M 186.403125 22.242454 \n","L 196.403125 22.242454 \n","L 206.403125 22.242454 \n","\" style=\"fill: none; stroke: #1f77b4; stroke-width: 1.5; stroke-linecap: square\"/>\n","    </g>\n","    <g id=\"text_11\">\n","     <!-- train -->\n","     <g transform=\"translate(214.403125 25.742454)scale(0.1 -0.1)\">\n","      <defs>\n","       <path id=\"DejaVuSans-74\" d=\"M 1172 4494 \n","L 1172 3500 \n","L 2356 3500 \n","L 2356 3053 \n","L 1172 3053 \n","L 1172 1153 \n","Q 1172 725 1289 603 \n","Q 1406 481 1766 481 \n","L 2356 481 \n","L 2356 0 \n","L 1766 0 \n","Q 1100 0 847 248 \n","Q 594 497 594 1153 \n","L 594 3053 \n","L 172 3053 \n","L 172 3500 \n","L 594 3500 \n","L 594 4494 \n","L 1172 4494 \n","z\n","\" transform=\"scale(0.015625)\"/>\n","       <path id=\"DejaVuSans-72\" d=\"M 2631 2963 \n","Q 2534 3019 2420 3045 \n","Q 2306 3072 2169 3072 \n","Q 1681 3072 1420 2755 \n","Q 1159 2438 1159 1844 \n","L 1159 0 \n","L 581 0 \n","L 581 3500 \n","L 1159 3500 \n","L 1159 2956 \n","Q 1341 3275 1631 3429 \n","Q 1922 3584 2338 3584 \n","Q 2397 3584 2469 3576 \n","Q 2541 3569 2628 3553 \n","L 2631 2963 \n","z\n","\" transform=\"scale(0.015625)\"/>\n","       <path id=\"DejaVuSans-61\" d=\"M 2194 1759 \n","Q 1497 1759 1228 1600 \n","Q 959 1441 959 1056 \n","Q 959 750 1161 570 \n","Q 1363 391 1709 391 \n","Q 2188 391 2477 730 \n","Q 2766 1069 2766 1631 \n","L 2766 1759 \n","L 2194 1759 \n","z\n","M 3341 1997 \n","L 3341 0 \n","L 2766 0 \n","L 2766 531 \n","Q 2569 213 2275 61 \n","Q 1981 -91 1556 -91 \n","Q 1019 -91 701 211 \n","Q 384 513 384 1019 \n","Q 384 1609 779 1909 \n","Q 1175 2209 1959 2209 \n","L 2766 2209 \n","L 2766 2266 \n","Q 2766 2663 2505 2880 \n","Q 2244 3097 1772 3097 \n","Q 1472 3097 1187 3025 \n","Q 903 2953 641 2809 \n","L 641 3341 \n","Q 956 3463 1253 3523 \n","Q 1550 3584 1831 3584 \n","Q 2591 3584 2966 3190 \n","Q 3341 2797 3341 1997 \n","z\n","\" transform=\"scale(0.015625)\"/>\n","       <path id=\"DejaVuSans-69\" d=\"M 603 3500 \n","L 1178 3500 \n","L 1178 0 \n","L 603 0 \n","L 603 3500 \n","z\n","M 603 4863 \n","L 1178 4863 \n","L 1178 4134 \n","L 603 4134 \n","L 603 4863 \n","z\n","\" transform=\"scale(0.015625)\"/>\n","       <path id=\"DejaVuSans-6e\" d=\"M 3513 2113 \n","L 3513 0 \n","L 2938 0 \n","L 2938 2094 \n","Q 2938 2591 2744 2837 \n","Q 2550 3084 2163 3084 \n","Q 1697 3084 1428 2787 \n","Q 1159 2491 1159 1978 \n","L 1159 0 \n","L 581 0 \n","L 581 3500 \n","L 1159 3500 \n","L 1159 2956 \n","Q 1366 3272 1645 3428 \n","Q 1925 3584 2291 3584 \n","Q 2894 3584 3203 3211 \n","Q 3513 2838 3513 2113 \n","z\n","\" transform=\"scale(0.015625)\"/>\n","      </defs>\n","      <use xlink:href=\"#DejaVuSans-74\"/>\n","      <use xlink:href=\"#DejaVuSans-72\" x=\"39.208984\"/>\n","      <use xlink:href=\"#DejaVuSans-61\" x=\"80.322266\"/>\n","      <use xlink:href=\"#DejaVuSans-69\" x=\"141.601562\"/>\n","      <use xlink:href=\"#DejaVuSans-6e\" x=\"169.384766\"/>\n","     </g>\n","    </g>\n","    <g id=\"line2d_20\">\n","     <path d=\"M 186.403125 36.920579 \n","L 196.403125 36.920579 \n","L 206.403125 36.920579 \n","\" style=\"fill: none; stroke-dasharray: 5.55,2.4; stroke-dashoffset: 0; stroke: #bf00bf; stroke-width: 1.5\"/>\n","    </g>\n","    <g id=\"text_12\">\n","     <!-- test -->\n","     <g transform=\"translate(214.403125 40.420579)scale(0.1 -0.1)\">\n","      <use xlink:href=\"#DejaVuSans-74\"/>\n","      <use xlink:href=\"#DejaVuSans-65\" x=\"39.208984\"/>\n","      <use xlink:href=\"#DejaVuSans-73\" x=\"100.732422\"/>\n","      <use xlink:href=\"#DejaVuSans-74\" x=\"152.832031\"/>\n","     </g>\n","    </g>\n","   </g>\n","  </g>\n"," </g>\n"," <defs>\n","  <clipPath id=\"pd83c5e35c7\">\n","   <rect x=\"51.378125\" y=\"9.144016\" width=\"195.3\" height=\"135.9\"/>\n","  </clipPath>\n"," </defs>\n","</svg>\n"],"text/plain":["<Figure size 252x180 with 1 Axes>"]},"metadata":{"needs_background":"light"},"output_type":"display_data"}],"source":["train(lambd=3)"],"id":"87af15b2"},{"cell_type":"markdown","metadata":{"origin_pos":22,"id":"ba9edf3b"},"source":["## [**简洁实现**]\n","\n","由于权重衰减在神经网络优化中很常用，\n","深度学习框架为了便于我们使用权重衰减，\n","将权重衰减集成到优化算法中，以便与任何损失函数结合使用。\n","此外，这种集成还有计算上的好处，\n","允许在不增加任何额外的计算开销的情况下向算法中添加权重衰减。\n","由于更新的权重衰减部分仅依赖于每个参数的当前值，\n","因此优化器必须至少接触每个参数一次。\n"],"id":"ba9edf3b"},{"cell_type":"markdown","metadata":{"origin_pos":24,"tab":["pytorch"],"id":"06225473"},"source":["在下面的代码中，我们在实例化优化器时直接通过`weight_decay`指定weight decay超参数。\n","默认情况下，PyTorch同时衰减权重和偏移。\n","这里我们只为权重设置了`weight_decay`，所以偏置参数$b$不会衰减。\n"],"id":"06225473"},{"cell_type":"code","execution_count":null,"metadata":{"execution":{"iopub.execute_input":"2022-07-31T02:23:01.983489Z","iopub.status.busy":"2022-07-31T02:23:01.982884Z","iopub.status.idle":"2022-07-31T02:23:01.991713Z","shell.execute_reply":"2022-07-31T02:23:01.990937Z"},"origin_pos":27,"tab":["pytorch"],"id":"ceed25eb"},"outputs":[],"source":["def train_concise(wd):\n","    net = nn.Sequential(nn.Linear(num_inputs, 1))\n","    for param in net.parameters():\n","        param.data.normal_()\n","    loss = nn.MSELoss(reduction='none')\n","    num_epochs, lr = 100, 0.003\n","    # 偏置参数没有衰减\n","    trainer = torch.optim.SGD([\n","        {\"params\":net[0].weight,'weight_decay': wd},\n","        {\"params\":net[0].bias}], lr=lr)\n","    animator = d2l.Animator(xlabel='epochs', ylabel='loss', yscale='log',\n","                            xlim=[5, num_epochs], legend=['train', 'test'])\n","    for epoch in range(num_epochs):\n","        for X, y in train_iter:\n","            trainer.zero_grad()\n","            l = loss(net(X), y)\n","            l.mean().backward()\n","            trainer.step()\n","        if (epoch + 1) % 5 == 0:\n","            animator.add(epoch + 1,\n","                         (d2l.evaluate_loss(net, train_iter, loss),\n","                          d2l.evaluate_loss(net, test_iter, loss)))\n","    print('w的L2范数：', net[0].weight.norm().item())"],"id":"ceed25eb"},{"cell_type":"markdown","metadata":{"origin_pos":29,"id":"4a00b6d6"},"source":["[**这些图看起来和我们从零开始实现权重衰减时的图相同**]。\n","然而，它们运行得更快，更容易实现。\n","对于更复杂的问题，这一好处将变得更加明显。\n"],"id":"4a00b6d6"},{"cell_type":"code","execution_count":null,"metadata":{"execution":{"iopub.execute_input":"2022-07-31T02:23:01.995096Z","iopub.status.busy":"2022-07-31T02:23:01.994625Z","iopub.status.idle":"2022-07-31T02:23:07.965364Z","shell.execute_reply":"2022-07-31T02:23:07.964719Z"},"origin_pos":30,"tab":["pytorch"],"id":"ba931419","outputId":"20639d14-5066-4be4-ba3e-3518fc97ee92"},"outputs":[{"name":"stdout","output_type":"stream","text":["w的L2范数： 14.026016235351562\n"]},{"data":{"image/svg+xml":["<?xml version=\"1.0\" encoding=\"utf-8\" standalone=\"no\"?>\n","<!DOCTYPE svg PUBLIC \"-//W3C//DTD SVG 1.1//EN\"\n","  \"http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd\">\n","<svg xmlns:xlink=\"http://www.w3.org/1999/xlink\" width=\"263.421875pt\" height=\"180.65625pt\" viewBox=\"0 0 263.421875 180.65625\" xmlns=\"http://www.w3.org/2000/svg\" version=\"1.1\">\n"," <metadata>\n","  <rdf:RDF xmlns:dc=\"http://purl.org/dc/elements/1.1/\" xmlns:cc=\"http://creativecommons.org/ns#\" xmlns:rdf=\"http://www.w3.org/1999/02/22-rdf-syntax-ns#\">\n","   <cc:Work>\n","    <dc:type rdf:resource=\"http://purl.org/dc/dcmitype/StillImage\"/>\n","    <dc:date>2022-07-31T02:23:07.884250</dc:date>\n","    <dc:format>image/svg+xml</dc:format>\n","    <dc:creator>\n","     <cc:Agent>\n","      <dc:title>Matplotlib v3.5.1, https://matplotlib.org/</dc:title>\n","     </cc:Agent>\n","    </dc:creator>\n","   </cc:Work>\n","  </rdf:RDF>\n"," </metadata>\n"," <defs>\n","  <style type=\"text/css\">*{stroke-linejoin: round; stroke-linecap: butt}</style>\n"," </defs>\n"," <g id=\"figure_1\">\n","  <g id=\"patch_1\">\n","   <path d=\"M 0 180.65625 \n","L 263.421875 180.65625 \n","L 263.421875 0 \n","L 0 0 \n","L 0 180.65625 \n","z\n","\" style=\"fill: none\"/>\n","  </g>\n","  <g id=\"axes_1\">\n","   <g id=\"patch_2\">\n","    <path d=\"M 51.378125 143.1 \n","L 246.678125 143.1 \n","L 246.678125 7.2 \n","L 51.378125 7.2 \n","z\n","\" style=\"fill: #ffffff\"/>\n","   </g>\n","   <g id=\"matplotlib.axis_1\">\n","    <g id=\"xtick_1\">\n","     <g id=\"line2d_1\">\n","      <path d=\"M 82.214967 143.1 \n","L 82.214967 7.2 \n","\" clip-path=\"url(#p5ed042e645)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n","     </g>\n","     <g id=\"line2d_2\">\n","      <defs>\n","       <path id=\"me20a397f43\" d=\"M 0 0 \n","L 0 3.5 \n","\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n","      </defs>\n","      <g>\n","       <use xlink:href=\"#me20a397f43\" x=\"82.214967\" y=\"143.1\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n","      </g>\n","     </g>\n","     <g id=\"text_1\">\n","      <!-- 20 -->\n","      <g transform=\"translate(75.852467 157.698438)scale(0.1 -0.1)\">\n","       <defs>\n","        <path id=\"DejaVuSans-32\" d=\"M 1228 531 \n","L 3431 531 \n","L 3431 0 \n","L 469 0 \n","L 469 531 \n","Q 828 903 1448 1529 \n","Q 2069 2156 2228 2338 \n","Q 2531 2678 2651 2914 \n","Q 2772 3150 2772 3378 \n","Q 2772 3750 2511 3984 \n","Q 2250 4219 1831 4219 \n","Q 1534 4219 1204 4116 \n","Q 875 4013 500 3803 \n","L 500 4441 \n","Q 881 4594 1212 4672 \n","Q 1544 4750 1819 4750 \n","Q 2544 4750 2975 4387 \n","Q 3406 4025 3406 3419 \n","Q 3406 3131 3298 2873 \n","Q 3191 2616 2906 2266 \n","Q 2828 2175 2409 1742 \n","Q 1991 1309 1228 531 \n","z\n","\" transform=\"scale(0.015625)\"/>\n","        <path id=\"DejaVuSans-30\" d=\"M 2034 4250 \n","Q 1547 4250 1301 3770 \n","Q 1056 3291 1056 2328 \n","Q 1056 1369 1301 889 \n","Q 1547 409 2034 409 \n","Q 2525 409 2770 889 \n","Q 3016 1369 3016 2328 \n","Q 3016 3291 2770 3770 \n","Q 2525 4250 2034 4250 \n","z\n","M 2034 4750 \n","Q 2819 4750 3233 4129 \n","Q 3647 3509 3647 2328 \n","Q 3647 1150 3233 529 \n","Q 2819 -91 2034 -91 \n","Q 1250 -91 836 529 \n","Q 422 1150 422 2328 \n","Q 422 3509 836 4129 \n","Q 1250 4750 2034 4750 \n","z\n","\" transform=\"scale(0.015625)\"/>\n","       </defs>\n","       <use xlink:href=\"#DejaVuSans-32\"/>\n","       <use xlink:href=\"#DejaVuSans-30\" x=\"63.623047\"/>\n","      </g>\n","     </g>\n","    </g>\n","    <g id=\"xtick_2\">\n","     <g id=\"line2d_3\">\n","      <path d=\"M 123.330757 143.1 \n","L 123.330757 7.2 \n","\" clip-path=\"url(#p5ed042e645)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n","     </g>\n","     <g id=\"line2d_4\">\n","      <g>\n","       <use xlink:href=\"#me20a397f43\" x=\"123.330757\" y=\"143.1\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n","      </g>\n","     </g>\n","     <g id=\"text_2\">\n","      <!-- 40 -->\n","      <g transform=\"translate(116.968257 157.698438)scale(0.1 -0.1)\">\n","       <defs>\n","        <path id=\"DejaVuSans-34\" d=\"M 2419 4116 \n","L 825 1625 \n","L 2419 1625 \n","L 2419 4116 \n","z\n","M 2253 4666 \n","L 3047 4666 \n","L 3047 1625 \n","L 3713 1625 \n","L 3713 1100 \n","L 3047 1100 \n","L 3047 0 \n","L 2419 0 \n","L 2419 1100 \n","L 313 1100 \n","L 313 1709 \n","L 2253 4666 \n","z\n","\" transform=\"scale(0.015625)\"/>\n","       </defs>\n","       <use xlink:href=\"#DejaVuSans-34\"/>\n","       <use xlink:href=\"#DejaVuSans-30\" x=\"63.623047\"/>\n","      </g>\n","     </g>\n","    </g>\n","    <g id=\"xtick_3\">\n","     <g id=\"line2d_5\">\n","      <path d=\"M 164.446546 143.1 \n","L 164.446546 7.2 \n","\" clip-path=\"url(#p5ed042e645)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n","     </g>\n","     <g id=\"line2d_6\">\n","      <g>\n","       <use xlink:href=\"#me20a397f43\" x=\"164.446546\" y=\"143.1\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n","      </g>\n","     </g>\n","     <g id=\"text_3\">\n","      <!-- 60 -->\n","      <g transform=\"translate(158.084046 157.698438)scale(0.1 -0.1)\">\n","       <defs>\n","        <path id=\"DejaVuSans-36\" d=\"M 2113 2584 \n","Q 1688 2584 1439 2293 \n","Q 1191 2003 1191 1497 \n","Q 1191 994 1439 701 \n","Q 1688 409 2113 409 \n","Q 2538 409 2786 701 \n","Q 3034 994 3034 1497 \n","Q 3034 2003 2786 2293 \n","Q 2538 2584 2113 2584 \n","z\n","M 3366 4563 \n","L 3366 3988 \n","Q 3128 4100 2886 4159 \n","Q 2644 4219 2406 4219 \n","Q 1781 4219 1451 3797 \n","Q 1122 3375 1075 2522 \n","Q 1259 2794 1537 2939 \n","Q 1816 3084 2150 3084 \n","Q 2853 3084 3261 2657 \n","Q 3669 2231 3669 1497 \n","Q 3669 778 3244 343 \n","Q 2819 -91 2113 -91 \n","Q 1303 -91 875 529 \n","Q 447 1150 447 2328 \n","Q 447 3434 972 4092 \n","Q 1497 4750 2381 4750 \n","Q 2619 4750 2861 4703 \n","Q 3103 4656 3366 4563 \n","z\n","\" transform=\"scale(0.015625)\"/>\n","       </defs>\n","       <use xlink:href=\"#DejaVuSans-36\"/>\n","       <use xlink:href=\"#DejaVuSans-30\" x=\"63.623047\"/>\n","      </g>\n","     </g>\n","    </g>\n","    <g id=\"xtick_4\">\n","     <g id=\"line2d_7\">\n","      <path d=\"M 205.562336 143.1 \n","L 205.562336 7.2 \n","\" clip-path=\"url(#p5ed042e645)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n","     </g>\n","     <g id=\"line2d_8\">\n","      <g>\n","       <use xlink:href=\"#me20a397f43\" x=\"205.562336\" y=\"143.1\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n","      </g>\n","     </g>\n","     <g id=\"text_4\">\n","      <!-- 80 -->\n","      <g transform=\"translate(199.199836 157.698438)scale(0.1 -0.1)\">\n","       <defs>\n","        <path id=\"DejaVuSans-38\" d=\"M 2034 2216 \n","Q 1584 2216 1326 1975 \n","Q 1069 1734 1069 1313 \n","Q 1069 891 1326 650 \n","Q 1584 409 2034 409 \n","Q 2484 409 2743 651 \n","Q 3003 894 3003 1313 \n","Q 3003 1734 2745 1975 \n","Q 2488 2216 2034 2216 \n","z\n","M 1403 2484 \n","Q 997 2584 770 2862 \n","Q 544 3141 544 3541 \n","Q 544 4100 942 4425 \n","Q 1341 4750 2034 4750 \n","Q 2731 4750 3128 4425 \n","Q 3525 4100 3525 3541 \n","Q 3525 3141 3298 2862 \n","Q 3072 2584 2669 2484 \n","Q 3125 2378 3379 2068 \n","Q 3634 1759 3634 1313 \n","Q 3634 634 3220 271 \n","Q 2806 -91 2034 -91 \n","Q 1263 -91 848 271 \n","Q 434 634 434 1313 \n","Q 434 1759 690 2068 \n","Q 947 2378 1403 2484 \n","z\n","M 1172 3481 \n","Q 1172 3119 1398 2916 \n","Q 1625 2713 2034 2713 \n","Q 2441 2713 2670 2916 \n","Q 2900 3119 2900 3481 \n","Q 2900 3844 2670 4047 \n","Q 2441 4250 2034 4250 \n","Q 1625 4250 1398 4047 \n","Q 1172 3844 1172 3481 \n","z\n","\" transform=\"scale(0.015625)\"/>\n","       </defs>\n","       <use xlink:href=\"#DejaVuSans-38\"/>\n","       <use xlink:href=\"#DejaVuSans-30\" x=\"63.623047\"/>\n","      </g>\n","     </g>\n","    </g>\n","    <g id=\"xtick_5\">\n","     <g id=\"line2d_9\">\n","      <path d=\"M 246.678125 143.1 \n","L 246.678125 7.2 \n","\" clip-path=\"url(#p5ed042e645)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n","     </g>\n","     <g id=\"line2d_10\">\n","      <g>\n","       <use xlink:href=\"#me20a397f43\" x=\"246.678125\" y=\"143.1\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n","      </g>\n","     </g>\n","     <g id=\"text_5\">\n","      <!-- 100 -->\n","      <g transform=\"translate(237.134375 157.698438)scale(0.1 -0.1)\">\n","       <defs>\n","        <path id=\"DejaVuSans-31\" d=\"M 794 531 \n","L 1825 531 \n","L 1825 4091 \n","L 703 3866 \n","L 703 4441 \n","L 1819 4666 \n","L 2450 4666 \n","L 2450 531 \n","L 3481 531 \n","L 3481 0 \n","L 794 0 \n","L 794 531 \n","z\n","\" transform=\"scale(0.015625)\"/>\n","       </defs>\n","       <use xlink:href=\"#DejaVuSans-31\"/>\n","       <use xlink:href=\"#DejaVuSans-30\" x=\"63.623047\"/>\n","       <use xlink:href=\"#DejaVuSans-30\" x=\"127.246094\"/>\n","      </g>\n","     </g>\n","    </g>\n","    <g id=\"text_6\">\n","     <!-- epochs -->\n","     <g transform=\"translate(131.195312 171.376563)scale(0.1 -0.1)\">\n","      <defs>\n","       <path id=\"DejaVuSans-65\" d=\"M 3597 1894 \n","L 3597 1613 \n","L 953 1613 \n","Q 991 1019 1311 708 \n","Q 1631 397 2203 397 \n","Q 2534 397 2845 478 \n","Q 3156 559 3463 722 \n","L 3463 178 \n","Q 3153 47 2828 -22 \n","Q 2503 -91 2169 -91 \n","Q 1331 -91 842 396 \n","Q 353 884 353 1716 \n","Q 353 2575 817 3079 \n","Q 1281 3584 2069 3584 \n","Q 2775 3584 3186 3129 \n","Q 3597 2675 3597 1894 \n","z\n","M 3022 2063 \n","Q 3016 2534 2758 2815 \n","Q 2500 3097 2075 3097 \n","Q 1594 3097 1305 2825 \n","Q 1016 2553 972 2059 \n","L 3022 2063 \n","z\n","\" transform=\"scale(0.015625)\"/>\n","       <path id=\"DejaVuSans-70\" d=\"M 1159 525 \n","L 1159 -1331 \n","L 581 -1331 \n","L 581 3500 \n","L 1159 3500 \n","L 1159 2969 \n","Q 1341 3281 1617 3432 \n","Q 1894 3584 2278 3584 \n","Q 2916 3584 3314 3078 \n","Q 3713 2572 3713 1747 \n","Q 3713 922 3314 415 \n","Q 2916 -91 2278 -91 \n","Q 1894 -91 1617 61 \n","Q 1341 213 1159 525 \n","z\n","M 3116 1747 \n","Q 3116 2381 2855 2742 \n","Q 2594 3103 2138 3103 \n","Q 1681 3103 1420 2742 \n","Q 1159 2381 1159 1747 \n","Q 1159 1113 1420 752 \n","Q 1681 391 2138 391 \n","Q 2594 391 2855 752 \n","Q 3116 1113 3116 1747 \n","z\n","\" transform=\"scale(0.015625)\"/>\n","       <path id=\"DejaVuSans-6f\" d=\"M 1959 3097 \n","Q 1497 3097 1228 2736 \n","Q 959 2375 959 1747 \n","Q 959 1119 1226 758 \n","Q 1494 397 1959 397 \n","Q 2419 397 2687 759 \n","Q 2956 1122 2956 1747 \n","Q 2956 2369 2687 2733 \n","Q 2419 3097 1959 3097 \n","z\n","M 1959 3584 \n","Q 2709 3584 3137 3096 \n","Q 3566 2609 3566 1747 \n","Q 3566 888 3137 398 \n","Q 2709 -91 1959 -91 \n","Q 1206 -91 779 398 \n","Q 353 888 353 1747 \n","Q 353 2609 779 3096 \n","Q 1206 3584 1959 3584 \n","z\n","\" transform=\"scale(0.015625)\"/>\n","       <path id=\"DejaVuSans-63\" d=\"M 3122 3366 \n","L 3122 2828 \n","Q 2878 2963 2633 3030 \n","Q 2388 3097 2138 3097 \n","Q 1578 3097 1268 2742 \n","Q 959 2388 959 1747 \n","Q 959 1106 1268 751 \n","Q 1578 397 2138 397 \n","Q 2388 397 2633 464 \n","Q 2878 531 3122 666 \n","L 3122 134 \n","Q 2881 22 2623 -34 \n","Q 2366 -91 2075 -91 \n","Q 1284 -91 818 406 \n","Q 353 903 353 1747 \n","Q 353 2603 823 3093 \n","Q 1294 3584 2113 3584 \n","Q 2378 3584 2631 3529 \n","Q 2884 3475 3122 3366 \n","z\n","\" transform=\"scale(0.015625)\"/>\n","       <path id=\"DejaVuSans-68\" d=\"M 3513 2113 \n","L 3513 0 \n","L 2938 0 \n","L 2938 2094 \n","Q 2938 2591 2744 2837 \n","Q 2550 3084 2163 3084 \n","Q 1697 3084 1428 2787 \n","Q 1159 2491 1159 1978 \n","L 1159 0 \n","L 581 0 \n","L 581 4863 \n","L 1159 4863 \n","L 1159 2956 \n","Q 1366 3272 1645 3428 \n","Q 1925 3584 2291 3584 \n","Q 2894 3584 3203 3211 \n","Q 3513 2838 3513 2113 \n","z\n","\" transform=\"scale(0.015625)\"/>\n","       <path id=\"DejaVuSans-73\" d=\"M 2834 3397 \n","L 2834 2853 \n","Q 2591 2978 2328 3040 \n","Q 2066 3103 1784 3103 \n","Q 1356 3103 1142 2972 \n","Q 928 2841 928 2578 \n","Q 928 2378 1081 2264 \n","Q 1234 2150 1697 2047 \n","L 1894 2003 \n","Q 2506 1872 2764 1633 \n","Q 3022 1394 3022 966 \n","Q 3022 478 2636 193 \n","Q 2250 -91 1575 -91 \n","Q 1294 -91 989 -36 \n","Q 684 19 347 128 \n","L 347 722 \n","Q 666 556 975 473 \n","Q 1284 391 1588 391 \n","Q 1994 391 2212 530 \n","Q 2431 669 2431 922 \n","Q 2431 1156 2273 1281 \n","Q 2116 1406 1581 1522 \n","L 1381 1569 \n","Q 847 1681 609 1914 \n","Q 372 2147 372 2553 \n","Q 372 3047 722 3315 \n","Q 1072 3584 1716 3584 \n","Q 2034 3584 2315 3537 \n","Q 2597 3491 2834 3397 \n","z\n","\" transform=\"scale(0.015625)\"/>\n","      </defs>\n","      <use xlink:href=\"#DejaVuSans-65\"/>\n","      <use xlink:href=\"#DejaVuSans-70\" x=\"61.523438\"/>\n","      <use xlink:href=\"#DejaVuSans-6f\" x=\"125\"/>\n","      <use xlink:href=\"#DejaVuSans-63\" x=\"186.181641\"/>\n","      <use xlink:href=\"#DejaVuSans-68\" x=\"241.162109\"/>\n","      <use xlink:href=\"#DejaVuSans-73\" x=\"304.541016\"/>\n","     </g>\n","    </g>\n","   </g>\n","   <g id=\"matplotlib.axis_2\">\n","    <g id=\"ytick_1\">\n","     <g id=\"line2d_11\">\n","      <path d=\"M 51.378125 119.277003 \n","L 246.678125 119.277003 \n","\" clip-path=\"url(#p5ed042e645)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n","     </g>\n","     <g id=\"line2d_12\">\n","      <defs>\n","       <path id=\"mb887f30291\" d=\"M 0 0 \n","L -3.5 0 \n","\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n","      </defs>\n","      <g>\n","       <use xlink:href=\"#mb887f30291\" x=\"51.378125\" y=\"119.277003\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n","      </g>\n","     </g>\n","     <g id=\"text_7\">\n","      <!-- $\\mathdefault{10^{-9}}$ -->\n","      <g transform=\"translate(20.878125 123.076222)scale(0.1 -0.1)\">\n","       <defs>\n","        <path id=\"DejaVuSans-2212\" d=\"M 678 2272 \n","L 4684 2272 \n","L 4684 1741 \n","L 678 1741 \n","L 678 2272 \n","z\n","\" transform=\"scale(0.015625)\"/>\n","        <path id=\"DejaVuSans-39\" d=\"M 703 97 \n","L 703 672 \n","Q 941 559 1184 500 \n","Q 1428 441 1663 441 \n","Q 2288 441 2617 861 \n","Q 2947 1281 2994 2138 \n","Q 2813 1869 2534 1725 \n","Q 2256 1581 1919 1581 \n","Q 1219 1581 811 2004 \n","Q 403 2428 403 3163 \n","Q 403 3881 828 4315 \n","Q 1253 4750 1959 4750 \n","Q 2769 4750 3195 4129 \n","Q 3622 3509 3622 2328 \n","Q 3622 1225 3098 567 \n","Q 2575 -91 1691 -91 \n","Q 1453 -91 1209 -44 \n","Q 966 3 703 97 \n","z\n","M 1959 2075 \n","Q 2384 2075 2632 2365 \n","Q 2881 2656 2881 3163 \n","Q 2881 3666 2632 3958 \n","Q 2384 4250 1959 4250 \n","Q 1534 4250 1286 3958 \n","Q 1038 3666 1038 3163 \n","Q 1038 2656 1286 2365 \n","Q 1534 2075 1959 2075 \n","z\n","\" transform=\"scale(0.015625)\"/>\n","       </defs>\n","       <use xlink:href=\"#DejaVuSans-31\" transform=\"translate(0 0.765625)\"/>\n","       <use xlink:href=\"#DejaVuSans-30\" transform=\"translate(63.623047 0.765625)\"/>\n","       <use xlink:href=\"#DejaVuSans-2212\" transform=\"translate(128.203125 39.046875)scale(0.7)\"/>\n","       <use xlink:href=\"#DejaVuSans-39\" transform=\"translate(186.855469 39.046875)scale(0.7)\"/>\n","      </g>\n","     </g>\n","    </g>\n","    <g id=\"ytick_2\">\n","     <g id=\"line2d_13\">\n","      <path d=\"M 51.378125 91.178407 \n","L 246.678125 91.178407 \n","\" clip-path=\"url(#p5ed042e645)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n","     </g>\n","     <g id=\"line2d_14\">\n","      <g>\n","       <use xlink:href=\"#mb887f30291\" x=\"51.378125\" y=\"91.178407\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n","      </g>\n","     </g>\n","     <g id=\"text_8\">\n","      <!-- $\\mathdefault{10^{-6}}$ -->\n","      <g transform=\"translate(20.878125 94.977626)scale(0.1 -0.1)\">\n","       <use xlink:href=\"#DejaVuSans-31\" transform=\"translate(0 0.765625)\"/>\n","       <use xlink:href=\"#DejaVuSans-30\" transform=\"translate(63.623047 0.765625)\"/>\n","       <use xlink:href=\"#DejaVuSans-2212\" transform=\"translate(128.203125 39.046875)scale(0.7)\"/>\n","       <use xlink:href=\"#DejaVuSans-36\" transform=\"translate(186.855469 39.046875)scale(0.7)\"/>\n","      </g>\n","     </g>\n","    </g>\n","    <g id=\"ytick_3\">\n","     <g id=\"line2d_15\">\n","      <path d=\"M 51.378125 63.079811 \n","L 246.678125 63.079811 \n","\" clip-path=\"url(#p5ed042e645)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n","     </g>\n","     <g id=\"line2d_16\">\n","      <g>\n","       <use xlink:href=\"#mb887f30291\" x=\"51.378125\" y=\"63.079811\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n","      </g>\n","     </g>\n","     <g id=\"text_9\">\n","      <!-- $\\mathdefault{10^{-3}}$ -->\n","      <g transform=\"translate(20.878125 66.87903)scale(0.1 -0.1)\">\n","       <defs>\n","        <path id=\"DejaVuSans-33\" d=\"M 2597 2516 \n","Q 3050 2419 3304 2112 \n","Q 3559 1806 3559 1356 \n","Q 3559 666 3084 287 \n","Q 2609 -91 1734 -91 \n","Q 1441 -91 1130 -33 \n","Q 819 25 488 141 \n","L 488 750 \n","Q 750 597 1062 519 \n","Q 1375 441 1716 441 \n","Q 2309 441 2620 675 \n","Q 2931 909 2931 1356 \n","Q 2931 1769 2642 2001 \n","Q 2353 2234 1838 2234 \n","L 1294 2234 \n","L 1294 2753 \n","L 1863 2753 \n","Q 2328 2753 2575 2939 \n","Q 2822 3125 2822 3475 \n","Q 2822 3834 2567 4026 \n","Q 2313 4219 1838 4219 \n","Q 1578 4219 1281 4162 \n","Q 984 4106 628 3988 \n","L 628 4550 \n","Q 988 4650 1302 4700 \n","Q 1616 4750 1894 4750 \n","Q 2613 4750 3031 4423 \n","Q 3450 4097 3450 3541 \n","Q 3450 3153 3228 2886 \n","Q 3006 2619 2597 2516 \n","z\n","\" transform=\"scale(0.015625)\"/>\n","       </defs>\n","       <use xlink:href=\"#DejaVuSans-31\" transform=\"translate(0 0.765625)\"/>\n","       <use xlink:href=\"#DejaVuSans-30\" transform=\"translate(63.623047 0.765625)\"/>\n","       <use xlink:href=\"#DejaVuSans-2212\" transform=\"translate(128.203125 39.046875)scale(0.7)\"/>\n","       <use xlink:href=\"#DejaVuSans-33\" transform=\"translate(186.855469 39.046875)scale(0.7)\"/>\n","      </g>\n","     </g>\n","    </g>\n","    <g id=\"ytick_4\">\n","     <g id=\"line2d_17\">\n","      <path d=\"M 51.378125 34.981215 \n","L 246.678125 34.981215 \n","\" clip-path=\"url(#p5ed042e645)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n","     </g>\n","     <g id=\"line2d_18\">\n","      <g>\n","       <use xlink:href=\"#mb887f30291\" x=\"51.378125\" y=\"34.981215\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n","      </g>\n","     </g>\n","     <g id=\"text_10\">\n","      <!-- $\\mathdefault{10^{0}}$ -->\n","      <g transform=\"translate(26.778125 38.780434)scale(0.1 -0.1)\">\n","       <use xlink:href=\"#DejaVuSans-31\" transform=\"translate(0 0.765625)\"/>\n","       <use xlink:href=\"#DejaVuSans-30\" transform=\"translate(63.623047 0.765625)\"/>\n","       <use xlink:href=\"#DejaVuSans-30\" transform=\"translate(128.203125 39.046875)scale(0.7)\"/>\n","      </g>\n","     </g>\n","    </g>\n","    <g id=\"text_11\">\n","     <!-- loss -->\n","     <g transform=\"translate(14.798437 84.807812)rotate(-90)scale(0.1 -0.1)\">\n","      <defs>\n","       <path id=\"DejaVuSans-6c\" d=\"M 603 4863 \n","L 1178 4863 \n","L 1178 0 \n","L 603 0 \n","L 603 4863 \n","z\n","\" transform=\"scale(0.015625)\"/>\n","      </defs>\n","      <use xlink:href=\"#DejaVuSans-6c\"/>\n","      <use xlink:href=\"#DejaVuSans-6f\" x=\"27.783203\"/>\n","      <use xlink:href=\"#DejaVuSans-73\" x=\"88.964844\"/>\n","      <use xlink:href=\"#DejaVuSans-73\" x=\"141.064453\"/>\n","     </g>\n","    </g>\n","   </g>\n","   <g id=\"line2d_19\">\n","    <path d=\"M 51.378125 24.168093 \n","L 61.657072 33.711551 \n","L 71.93602 41.915027 \n","L 82.214967 49.524061 \n","L 92.493914 56.738217 \n","L 102.772862 63.769639 \n","L 113.051809 70.64042 \n","L 123.330757 77.402652 \n","L 133.609704 84.085355 \n","L 143.888651 90.66234 \n","L 154.167599 97.107542 \n","L 164.446546 103.43541 \n","L 174.725493 109.653939 \n","L 185.004441 115.661614 \n","L 195.283388 121.408282 \n","L 205.562336 126.776031 \n","L 215.841283 130.811171 \n","L 226.12023 133.648442 \n","L 236.399178 135.602606 \n","L 246.678125 136.922727 \n","\" clip-path=\"url(#p5ed042e645)\" style=\"fill: none; stroke: #1f77b4; stroke-width: 1.5; stroke-linecap: square\"/>\n","   </g>\n","   <g id=\"line2d_20\">\n","    <path d=\"M 51.378125 13.377273 \n","L 61.657072 13.425898 \n","L 71.93602 13.447246 \n","L 82.214967 13.456447 \n","L 92.493914 13.460139 \n","L 102.772862 13.461661 \n","L 113.051809 13.462259 \n","L 123.330757 13.462501 \n","L 133.609704 13.462598 \n","L 143.888651 13.462637 \n","L 154.167599 13.462652 \n","L 164.446546 13.462658 \n","L 174.725493 13.462661 \n","L 185.004441 13.462662 \n","L 195.283388 13.462662 \n","L 205.562336 13.462662 \n","L 215.841283 13.462662 \n","L 226.12023 13.462662 \n","L 236.399178 13.462662 \n","L 246.678125 13.462662 \n","\" clip-path=\"url(#p5ed042e645)\" style=\"fill: none; stroke-dasharray: 5.55,2.4; stroke-dashoffset: 0; stroke: #bf00bf; stroke-width: 1.5\"/>\n","   </g>\n","   <g id=\"patch_3\">\n","    <path d=\"M 51.378125 143.1 \n","L 51.378125 7.2 \n","\" style=\"fill: none; stroke: #000000; stroke-width: 0.8; stroke-linejoin: miter; stroke-linecap: square\"/>\n","   </g>\n","   <g id=\"patch_4\">\n","    <path d=\"M 246.678125 143.1 \n","L 246.678125 7.2 \n","\" style=\"fill: none; stroke: #000000; stroke-width: 0.8; stroke-linejoin: miter; stroke-linecap: square\"/>\n","   </g>\n","   <g id=\"patch_5\">\n","    <path d=\"M 51.378125 143.1 \n","L 246.678125 143.1 \n","\" style=\"fill: none; stroke: #000000; stroke-width: 0.8; stroke-linejoin: miter; stroke-linecap: square\"/>\n","   </g>\n","   <g id=\"patch_6\">\n","    <path d=\"M 51.378125 7.2 \n","L 246.678125 7.2 \n","\" style=\"fill: none; stroke: #000000; stroke-width: 0.8; stroke-linejoin: miter; stroke-linecap: square\"/>\n","   </g>\n","   <g id=\"legend_1\">\n","    <g id=\"patch_7\">\n","     <path d=\"M 58.378125 138.1 \n","L 113.653125 138.1 \n","Q 115.653125 138.1 115.653125 136.1 \n","L 115.653125 107.74375 \n","Q 115.653125 105.74375 113.653125 105.74375 \n","L 58.378125 105.74375 \n","Q 56.378125 105.74375 56.378125 107.74375 \n","L 56.378125 136.1 \n","Q 56.378125 138.1 58.378125 138.1 \n","z\n","\" style=\"fill: #ffffff; opacity: 0.8; stroke: #cccccc; stroke-linejoin: miter\"/>\n","    </g>\n","    <g id=\"line2d_21\">\n","     <path d=\"M 60.378125 113.842187 \n","L 70.378125 113.842187 \n","L 80.378125 113.842187 \n","\" style=\"fill: none; stroke: #1f77b4; stroke-width: 1.5; stroke-linecap: square\"/>\n","    </g>\n","    <g id=\"text_12\">\n","     <!-- train -->\n","     <g transform=\"translate(88.378125 117.342187)scale(0.1 -0.1)\">\n","      <defs>\n","       <path id=\"DejaVuSans-74\" d=\"M 1172 4494 \n","L 1172 3500 \n","L 2356 3500 \n","L 2356 3053 \n","L 1172 3053 \n","L 1172 1153 \n","Q 1172 725 1289 603 \n","Q 1406 481 1766 481 \n","L 2356 481 \n","L 2356 0 \n","L 1766 0 \n","Q 1100 0 847 248 \n","Q 594 497 594 1153 \n","L 594 3053 \n","L 172 3053 \n","L 172 3500 \n","L 594 3500 \n","L 594 4494 \n","L 1172 4494 \n","z\n","\" transform=\"scale(0.015625)\"/>\n","       <path id=\"DejaVuSans-72\" d=\"M 2631 2963 \n","Q 2534 3019 2420 3045 \n","Q 2306 3072 2169 3072 \n","Q 1681 3072 1420 2755 \n","Q 1159 2438 1159 1844 \n","L 1159 0 \n","L 581 0 \n","L 581 3500 \n","L 1159 3500 \n","L 1159 2956 \n","Q 1341 3275 1631 3429 \n","Q 1922 3584 2338 3584 \n","Q 2397 3584 2469 3576 \n","Q 2541 3569 2628 3553 \n","L 2631 2963 \n","z\n","\" transform=\"scale(0.015625)\"/>\n","       <path id=\"DejaVuSans-61\" d=\"M 2194 1759 \n","Q 1497 1759 1228 1600 \n","Q 959 1441 959 1056 \n","Q 959 750 1161 570 \n","Q 1363 391 1709 391 \n","Q 2188 391 2477 730 \n","Q 2766 1069 2766 1631 \n","L 2766 1759 \n","L 2194 1759 \n","z\n","M 3341 1997 \n","L 3341 0 \n","L 2766 0 \n","L 2766 531 \n","Q 2569 213 2275 61 \n","Q 1981 -91 1556 -91 \n","Q 1019 -91 701 211 \n","Q 384 513 384 1019 \n","Q 384 1609 779 1909 \n","Q 1175 2209 1959 2209 \n","L 2766 2209 \n","L 2766 2266 \n","Q 2766 2663 2505 2880 \n","Q 2244 3097 1772 3097 \n","Q 1472 3097 1187 3025 \n","Q 903 2953 641 2809 \n","L 641 3341 \n","Q 956 3463 1253 3523 \n","Q 1550 3584 1831 3584 \n","Q 2591 3584 2966 3190 \n","Q 3341 2797 3341 1997 \n","z\n","\" transform=\"scale(0.015625)\"/>\n","       <path id=\"DejaVuSans-69\" d=\"M 603 3500 \n","L 1178 3500 \n","L 1178 0 \n","L 603 0 \n","L 603 3500 \n","z\n","M 603 4863 \n","L 1178 4863 \n","L 1178 4134 \n","L 603 4134 \n","L 603 4863 \n","z\n","\" transform=\"scale(0.015625)\"/>\n","       <path id=\"DejaVuSans-6e\" d=\"M 3513 2113 \n","L 3513 0 \n","L 2938 0 \n","L 2938 2094 \n","Q 2938 2591 2744 2837 \n","Q 2550 3084 2163 3084 \n","Q 1697 3084 1428 2787 \n","Q 1159 2491 1159 1978 \n","L 1159 0 \n","L 581 0 \n","L 581 3500 \n","L 1159 3500 \n","L 1159 2956 \n","Q 1366 3272 1645 3428 \n","Q 1925 3584 2291 3584 \n","Q 2894 3584 3203 3211 \n","Q 3513 2838 3513 2113 \n","z\n","\" transform=\"scale(0.015625)\"/>\n","      </defs>\n","      <use xlink:href=\"#DejaVuSans-74\"/>\n","      <use xlink:href=\"#DejaVuSans-72\" x=\"39.208984\"/>\n","      <use xlink:href=\"#DejaVuSans-61\" x=\"80.322266\"/>\n","      <use xlink:href=\"#DejaVuSans-69\" x=\"141.601562\"/>\n","      <use xlink:href=\"#DejaVuSans-6e\" x=\"169.384766\"/>\n","     </g>\n","    </g>\n","    <g id=\"line2d_22\">\n","     <path d=\"M 60.378125 128.520312 \n","L 70.378125 128.520312 \n","L 80.378125 128.520312 \n","\" style=\"fill: none; stroke-dasharray: 5.55,2.4; stroke-dashoffset: 0; stroke: #bf00bf; stroke-width: 1.5\"/>\n","    </g>\n","    <g id=\"text_13\">\n","     <!-- test -->\n","     <g transform=\"translate(88.378125 132.020312)scale(0.1 -0.1)\">\n","      <use xlink:href=\"#DejaVuSans-74\"/>\n","      <use xlink:href=\"#DejaVuSans-65\" x=\"39.208984\"/>\n","      <use xlink:href=\"#DejaVuSans-73\" x=\"100.732422\"/>\n","      <use xlink:href=\"#DejaVuSans-74\" x=\"152.832031\"/>\n","     </g>\n","    </g>\n","   </g>\n","  </g>\n"," </g>\n"," <defs>\n","  <clipPath id=\"p5ed042e645\">\n","   <rect x=\"51.378125\" y=\"7.2\" width=\"195.3\" height=\"135.9\"/>\n","  </clipPath>\n"," </defs>\n","</svg>\n"],"text/plain":["<Figure size 252x180 with 1 Axes>"]},"metadata":{"needs_background":"light"},"output_type":"display_data"}],"source":["train_concise(0)"],"id":"ba931419"},{"cell_type":"code","execution_count":null,"metadata":{"execution":{"iopub.execute_input":"2022-07-31T02:23:07.968650Z","iopub.status.busy":"2022-07-31T02:23:07.968162Z","iopub.status.idle":"2022-07-31T02:23:17.124533Z","shell.execute_reply":"2022-07-31T02:23:17.123862Z"},"origin_pos":31,"tab":["pytorch"],"id":"30767598","outputId":"344001d5-8b4a-4ad7-98eb-ab0dd4d122d2"},"outputs":[{"name":"stdout","output_type":"stream","text":["w的L2范数： 0.7186794877052307\n"]},{"data":{"image/svg+xml":["<?xml version=\"1.0\" encoding=\"utf-8\" standalone=\"no\"?>\n","<!DOCTYPE svg PUBLIC \"-//W3C//DTD SVG 1.1//EN\"\n","  \"http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd\">\n","<svg xmlns:xlink=\"http://www.w3.org/1999/xlink\" width=\"263.421875pt\" height=\"180.65625pt\" viewBox=\"0 0 263.421875 180.65625\" xmlns=\"http://www.w3.org/2000/svg\" version=\"1.1\">\n"," <metadata>\n","  <rdf:RDF xmlns:dc=\"http://purl.org/dc/elements/1.1/\" xmlns:cc=\"http://creativecommons.org/ns#\" xmlns:rdf=\"http://www.w3.org/1999/02/22-rdf-syntax-ns#\">\n","   <cc:Work>\n","    <dc:type rdf:resource=\"http://purl.org/dc/dcmitype/StillImage\"/>\n","    <dc:date>2022-07-31T02:23:17.007647</dc:date>\n","    <dc:format>image/svg+xml</dc:format>\n","    <dc:creator>\n","     <cc:Agent>\n","      <dc:title>Matplotlib v3.5.1, https://matplotlib.org/</dc:title>\n","     </cc:Agent>\n","    </dc:creator>\n","   </cc:Work>\n","  </rdf:RDF>\n"," </metadata>\n"," <defs>\n","  <style type=\"text/css\">*{stroke-linejoin: round; stroke-linecap: butt}</style>\n"," </defs>\n"," <g id=\"figure_1\">\n","  <g id=\"patch_1\">\n","   <path d=\"M 0 180.65625 \n","L 263.421875 180.65625 \n","L 263.421875 0 \n","L 0 0 \n","L 0 180.65625 \n","z\n","\" style=\"fill: none\"/>\n","  </g>\n","  <g id=\"axes_1\">\n","   <g id=\"patch_2\">\n","    <path d=\"M 51.378125 143.1 \n","L 246.678125 143.1 \n","L 246.678125 7.2 \n","L 51.378125 7.2 \n","z\n","\" style=\"fill: #ffffff\"/>\n","   </g>\n","   <g id=\"matplotlib.axis_1\">\n","    <g id=\"xtick_1\">\n","     <g id=\"line2d_1\">\n","      <path d=\"M 82.214967 143.1 \n","L 82.214967 7.2 \n","\" clip-path=\"url(#p97ac790d64)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n","     </g>\n","     <g id=\"line2d_2\">\n","      <defs>\n","       <path id=\"m4856621ce4\" d=\"M 0 0 \n","L 0 3.5 \n","\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n","      </defs>\n","      <g>\n","       <use xlink:href=\"#m4856621ce4\" x=\"82.214967\" y=\"143.1\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n","      </g>\n","     </g>\n","     <g id=\"text_1\">\n","      <!-- 20 -->\n","      <g transform=\"translate(75.852467 157.698438)scale(0.1 -0.1)\">\n","       <defs>\n","        <path id=\"DejaVuSans-32\" d=\"M 1228 531 \n","L 3431 531 \n","L 3431 0 \n","L 469 0 \n","L 469 531 \n","Q 828 903 1448 1529 \n","Q 2069 2156 2228 2338 \n","Q 2531 2678 2651 2914 \n","Q 2772 3150 2772 3378 \n","Q 2772 3750 2511 3984 \n","Q 2250 4219 1831 4219 \n","Q 1534 4219 1204 4116 \n","Q 875 4013 500 3803 \n","L 500 4441 \n","Q 881 4594 1212 4672 \n","Q 1544 4750 1819 4750 \n","Q 2544 4750 2975 4387 \n","Q 3406 4025 3406 3419 \n","Q 3406 3131 3298 2873 \n","Q 3191 2616 2906 2266 \n","Q 2828 2175 2409 1742 \n","Q 1991 1309 1228 531 \n","z\n","\" transform=\"scale(0.015625)\"/>\n","        <path id=\"DejaVuSans-30\" d=\"M 2034 4250 \n","Q 1547 4250 1301 3770 \n","Q 1056 3291 1056 2328 \n","Q 1056 1369 1301 889 \n","Q 1547 409 2034 409 \n","Q 2525 409 2770 889 \n","Q 3016 1369 3016 2328 \n","Q 3016 3291 2770 3770 \n","Q 2525 4250 2034 4250 \n","z\n","M 2034 4750 \n","Q 2819 4750 3233 4129 \n","Q 3647 3509 3647 2328 \n","Q 3647 1150 3233 529 \n","Q 2819 -91 2034 -91 \n","Q 1250 -91 836 529 \n","Q 422 1150 422 2328 \n","Q 422 3509 836 4129 \n","Q 1250 4750 2034 4750 \n","z\n","\" transform=\"scale(0.015625)\"/>\n","       </defs>\n","       <use xlink:href=\"#DejaVuSans-32\"/>\n","       <use xlink:href=\"#DejaVuSans-30\" x=\"63.623047\"/>\n","      </g>\n","     </g>\n","    </g>\n","    <g id=\"xtick_2\">\n","     <g id=\"line2d_3\">\n","      <path d=\"M 123.330757 143.1 \n","L 123.330757 7.2 \n","\" clip-path=\"url(#p97ac790d64)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n","     </g>\n","     <g id=\"line2d_4\">\n","      <g>\n","       <use xlink:href=\"#m4856621ce4\" x=\"123.330757\" y=\"143.1\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n","      </g>\n","     </g>\n","     <g id=\"text_2\">\n","      <!-- 40 -->\n","      <g transform=\"translate(116.968257 157.698438)scale(0.1 -0.1)\">\n","       <defs>\n","        <path id=\"DejaVuSans-34\" d=\"M 2419 4116 \n","L 825 1625 \n","L 2419 1625 \n","L 2419 4116 \n","z\n","M 2253 4666 \n","L 3047 4666 \n","L 3047 1625 \n","L 3713 1625 \n","L 3713 1100 \n","L 3047 1100 \n","L 3047 0 \n","L 2419 0 \n","L 2419 1100 \n","L 313 1100 \n","L 313 1709 \n","L 2253 4666 \n","z\n","\" transform=\"scale(0.015625)\"/>\n","       </defs>\n","       <use xlink:href=\"#DejaVuSans-34\"/>\n","       <use xlink:href=\"#DejaVuSans-30\" x=\"63.623047\"/>\n","      </g>\n","     </g>\n","    </g>\n","    <g id=\"xtick_3\">\n","     <g id=\"line2d_5\">\n","      <path d=\"M 164.446546 143.1 \n","L 164.446546 7.2 \n","\" clip-path=\"url(#p97ac790d64)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n","     </g>\n","     <g id=\"line2d_6\">\n","      <g>\n","       <use xlink:href=\"#m4856621ce4\" x=\"164.446546\" y=\"143.1\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n","      </g>\n","     </g>\n","     <g id=\"text_3\">\n","      <!-- 60 -->\n","      <g transform=\"translate(158.084046 157.698438)scale(0.1 -0.1)\">\n","       <defs>\n","        <path id=\"DejaVuSans-36\" d=\"M 2113 2584 \n","Q 1688 2584 1439 2293 \n","Q 1191 2003 1191 1497 \n","Q 1191 994 1439 701 \n","Q 1688 409 2113 409 \n","Q 2538 409 2786 701 \n","Q 3034 994 3034 1497 \n","Q 3034 2003 2786 2293 \n","Q 2538 2584 2113 2584 \n","z\n","M 3366 4563 \n","L 3366 3988 \n","Q 3128 4100 2886 4159 \n","Q 2644 4219 2406 4219 \n","Q 1781 4219 1451 3797 \n","Q 1122 3375 1075 2522 \n","Q 1259 2794 1537 2939 \n","Q 1816 3084 2150 3084 \n","Q 2853 3084 3261 2657 \n","Q 3669 2231 3669 1497 \n","Q 3669 778 3244 343 \n","Q 2819 -91 2113 -91 \n","Q 1303 -91 875 529 \n","Q 447 1150 447 2328 \n","Q 447 3434 972 4092 \n","Q 1497 4750 2381 4750 \n","Q 2619 4750 2861 4703 \n","Q 3103 4656 3366 4563 \n","z\n","\" transform=\"scale(0.015625)\"/>\n","       </defs>\n","       <use xlink:href=\"#DejaVuSans-36\"/>\n","       <use xlink:href=\"#DejaVuSans-30\" x=\"63.623047\"/>\n","      </g>\n","     </g>\n","    </g>\n","    <g id=\"xtick_4\">\n","     <g id=\"line2d_7\">\n","      <path d=\"M 205.562336 143.1 \n","L 205.562336 7.2 \n","\" clip-path=\"url(#p97ac790d64)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n","     </g>\n","     <g id=\"line2d_8\">\n","      <g>\n","       <use xlink:href=\"#m4856621ce4\" x=\"205.562336\" y=\"143.1\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n","      </g>\n","     </g>\n","     <g id=\"text_4\">\n","      <!-- 80 -->\n","      <g transform=\"translate(199.199836 157.698438)scale(0.1 -0.1)\">\n","       <defs>\n","        <path id=\"DejaVuSans-38\" d=\"M 2034 2216 \n","Q 1584 2216 1326 1975 \n","Q 1069 1734 1069 1313 \n","Q 1069 891 1326 650 \n","Q 1584 409 2034 409 \n","Q 2484 409 2743 651 \n","Q 3003 894 3003 1313 \n","Q 3003 1734 2745 1975 \n","Q 2488 2216 2034 2216 \n","z\n","M 1403 2484 \n","Q 997 2584 770 2862 \n","Q 544 3141 544 3541 \n","Q 544 4100 942 4425 \n","Q 1341 4750 2034 4750 \n","Q 2731 4750 3128 4425 \n","Q 3525 4100 3525 3541 \n","Q 3525 3141 3298 2862 \n","Q 3072 2584 2669 2484 \n","Q 3125 2378 3379 2068 \n","Q 3634 1759 3634 1313 \n","Q 3634 634 3220 271 \n","Q 2806 -91 2034 -91 \n","Q 1263 -91 848 271 \n","Q 434 634 434 1313 \n","Q 434 1759 690 2068 \n","Q 947 2378 1403 2484 \n","z\n","M 1172 3481 \n","Q 1172 3119 1398 2916 \n","Q 1625 2713 2034 2713 \n","Q 2441 2713 2670 2916 \n","Q 2900 3119 2900 3481 \n","Q 2900 3844 2670 4047 \n","Q 2441 4250 2034 4250 \n","Q 1625 4250 1398 4047 \n","Q 1172 3844 1172 3481 \n","z\n","\" transform=\"scale(0.015625)\"/>\n","       </defs>\n","       <use xlink:href=\"#DejaVuSans-38\"/>\n","       <use xlink:href=\"#DejaVuSans-30\" x=\"63.623047\"/>\n","      </g>\n","     </g>\n","    </g>\n","    <g id=\"xtick_5\">\n","     <g id=\"line2d_9\">\n","      <path d=\"M 246.678125 143.1 \n","L 246.678125 7.2 \n","\" clip-path=\"url(#p97ac790d64)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n","     </g>\n","     <g id=\"line2d_10\">\n","      <g>\n","       <use xlink:href=\"#m4856621ce4\" x=\"246.678125\" y=\"143.1\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n","      </g>\n","     </g>\n","     <g id=\"text_5\">\n","      <!-- 100 -->\n","      <g transform=\"translate(237.134375 157.698438)scale(0.1 -0.1)\">\n","       <defs>\n","        <path id=\"DejaVuSans-31\" d=\"M 794 531 \n","L 1825 531 \n","L 1825 4091 \n","L 703 3866 \n","L 703 4441 \n","L 1819 4666 \n","L 2450 4666 \n","L 2450 531 \n","L 3481 531 \n","L 3481 0 \n","L 794 0 \n","L 794 531 \n","z\n","\" transform=\"scale(0.015625)\"/>\n","       </defs>\n","       <use xlink:href=\"#DejaVuSans-31\"/>\n","       <use xlink:href=\"#DejaVuSans-30\" x=\"63.623047\"/>\n","       <use xlink:href=\"#DejaVuSans-30\" x=\"127.246094\"/>\n","      </g>\n","     </g>\n","    </g>\n","    <g id=\"text_6\">\n","     <!-- epochs -->\n","     <g transform=\"translate(131.195312 171.376563)scale(0.1 -0.1)\">\n","      <defs>\n","       <path id=\"DejaVuSans-65\" d=\"M 3597 1894 \n","L 3597 1613 \n","L 953 1613 \n","Q 991 1019 1311 708 \n","Q 1631 397 2203 397 \n","Q 2534 397 2845 478 \n","Q 3156 559 3463 722 \n","L 3463 178 \n","Q 3153 47 2828 -22 \n","Q 2503 -91 2169 -91 \n","Q 1331 -91 842 396 \n","Q 353 884 353 1716 \n","Q 353 2575 817 3079 \n","Q 1281 3584 2069 3584 \n","Q 2775 3584 3186 3129 \n","Q 3597 2675 3597 1894 \n","z\n","M 3022 2063 \n","Q 3016 2534 2758 2815 \n","Q 2500 3097 2075 3097 \n","Q 1594 3097 1305 2825 \n","Q 1016 2553 972 2059 \n","L 3022 2063 \n","z\n","\" transform=\"scale(0.015625)\"/>\n","       <path id=\"DejaVuSans-70\" d=\"M 1159 525 \n","L 1159 -1331 \n","L 581 -1331 \n","L 581 3500 \n","L 1159 3500 \n","L 1159 2969 \n","Q 1341 3281 1617 3432 \n","Q 1894 3584 2278 3584 \n","Q 2916 3584 3314 3078 \n","Q 3713 2572 3713 1747 \n","Q 3713 922 3314 415 \n","Q 2916 -91 2278 -91 \n","Q 1894 -91 1617 61 \n","Q 1341 213 1159 525 \n","z\n","M 3116 1747 \n","Q 3116 2381 2855 2742 \n","Q 2594 3103 2138 3103 \n","Q 1681 3103 1420 2742 \n","Q 1159 2381 1159 1747 \n","Q 1159 1113 1420 752 \n","Q 1681 391 2138 391 \n","Q 2594 391 2855 752 \n","Q 3116 1113 3116 1747 \n","z\n","\" transform=\"scale(0.015625)\"/>\n","       <path id=\"DejaVuSans-6f\" d=\"M 1959 3097 \n","Q 1497 3097 1228 2736 \n","Q 959 2375 959 1747 \n","Q 959 1119 1226 758 \n","Q 1494 397 1959 397 \n","Q 2419 397 2687 759 \n","Q 2956 1122 2956 1747 \n","Q 2956 2369 2687 2733 \n","Q 2419 3097 1959 3097 \n","z\n","M 1959 3584 \n","Q 2709 3584 3137 3096 \n","Q 3566 2609 3566 1747 \n","Q 3566 888 3137 398 \n","Q 2709 -91 1959 -91 \n","Q 1206 -91 779 398 \n","Q 353 888 353 1747 \n","Q 353 2609 779 3096 \n","Q 1206 3584 1959 3584 \n","z\n","\" transform=\"scale(0.015625)\"/>\n","       <path id=\"DejaVuSans-63\" d=\"M 3122 3366 \n","L 3122 2828 \n","Q 2878 2963 2633 3030 \n","Q 2388 3097 2138 3097 \n","Q 1578 3097 1268 2742 \n","Q 959 2388 959 1747 \n","Q 959 1106 1268 751 \n","Q 1578 397 2138 397 \n","Q 2388 397 2633 464 \n","Q 2878 531 3122 666 \n","L 3122 134 \n","Q 2881 22 2623 -34 \n","Q 2366 -91 2075 -91 \n","Q 1284 -91 818 406 \n","Q 353 903 353 1747 \n","Q 353 2603 823 3093 \n","Q 1294 3584 2113 3584 \n","Q 2378 3584 2631 3529 \n","Q 2884 3475 3122 3366 \n","z\n","\" transform=\"scale(0.015625)\"/>\n","       <path id=\"DejaVuSans-68\" d=\"M 3513 2113 \n","L 3513 0 \n","L 2938 0 \n","L 2938 2094 \n","Q 2938 2591 2744 2837 \n","Q 2550 3084 2163 3084 \n","Q 1697 3084 1428 2787 \n","Q 1159 2491 1159 1978 \n","L 1159 0 \n","L 581 0 \n","L 581 4863 \n","L 1159 4863 \n","L 1159 2956 \n","Q 1366 3272 1645 3428 \n","Q 1925 3584 2291 3584 \n","Q 2894 3584 3203 3211 \n","Q 3513 2838 3513 2113 \n","z\n","\" transform=\"scale(0.015625)\"/>\n","       <path id=\"DejaVuSans-73\" d=\"M 2834 3397 \n","L 2834 2853 \n","Q 2591 2978 2328 3040 \n","Q 2066 3103 1784 3103 \n","Q 1356 3103 1142 2972 \n","Q 928 2841 928 2578 \n","Q 928 2378 1081 2264 \n","Q 1234 2150 1697 2047 \n","L 1894 2003 \n","Q 2506 1872 2764 1633 \n","Q 3022 1394 3022 966 \n","Q 3022 478 2636 193 \n","Q 2250 -91 1575 -91 \n","Q 1294 -91 989 -36 \n","Q 684 19 347 128 \n","L 347 722 \n","Q 666 556 975 473 \n","Q 1284 391 1588 391 \n","Q 1994 391 2212 530 \n","Q 2431 669 2431 922 \n","Q 2431 1156 2273 1281 \n","Q 2116 1406 1581 1522 \n","L 1381 1569 \n","Q 847 1681 609 1914 \n","Q 372 2147 372 2553 \n","Q 372 3047 722 3315 \n","Q 1072 3584 1716 3584 \n","Q 2034 3584 2315 3537 \n","Q 2597 3491 2834 3397 \n","z\n","\" transform=\"scale(0.015625)\"/>\n","      </defs>\n","      <use xlink:href=\"#DejaVuSans-65\"/>\n","      <use xlink:href=\"#DejaVuSans-70\" x=\"61.523438\"/>\n","      <use xlink:href=\"#DejaVuSans-6f\" x=\"125\"/>\n","      <use xlink:href=\"#DejaVuSans-63\" x=\"186.181641\"/>\n","      <use xlink:href=\"#DejaVuSans-68\" x=\"241.162109\"/>\n","      <use xlink:href=\"#DejaVuSans-73\" x=\"304.541016\"/>\n","     </g>\n","    </g>\n","   </g>\n","   <g id=\"matplotlib.axis_2\">\n","    <g id=\"ytick_1\">\n","     <g id=\"line2d_11\">\n","      <path d=\"M 51.378125 131.054124 \n","L 246.678125 131.054124 \n","\" clip-path=\"url(#p97ac790d64)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n","     </g>\n","     <g id=\"line2d_12\">\n","      <defs>\n","       <path id=\"m5d32c10c90\" d=\"M 0 0 \n","L -3.5 0 \n","\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n","      </defs>\n","      <g>\n","       <use xlink:href=\"#m5d32c10c90\" x=\"51.378125\" y=\"131.054124\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n","      </g>\n","     </g>\n","     <g id=\"text_7\">\n","      <!-- $\\mathdefault{10^{-1}}$ -->\n","      <g transform=\"translate(20.878125 134.853343)scale(0.1 -0.1)\">\n","       <defs>\n","        <path id=\"DejaVuSans-2212\" d=\"M 678 2272 \n","L 4684 2272 \n","L 4684 1741 \n","L 678 1741 \n","L 678 2272 \n","z\n","\" transform=\"scale(0.015625)\"/>\n","       </defs>\n","       <use xlink:href=\"#DejaVuSans-31\" transform=\"translate(0 0.684375)\"/>\n","       <use xlink:href=\"#DejaVuSans-30\" transform=\"translate(63.623047 0.684375)\"/>\n","       <use xlink:href=\"#DejaVuSans-2212\" transform=\"translate(128.203125 38.965625)scale(0.7)\"/>\n","       <use xlink:href=\"#DejaVuSans-31\" transform=\"translate(186.855469 38.965625)scale(0.7)\"/>\n","      </g>\n","     </g>\n","    </g>\n","    <g id=\"ytick_2\">\n","     <g id=\"line2d_13\">\n","      <path d=\"M 51.378125 95.094219 \n","L 246.678125 95.094219 \n","\" clip-path=\"url(#p97ac790d64)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n","     </g>\n","     <g id=\"line2d_14\">\n","      <g>\n","       <use xlink:href=\"#m5d32c10c90\" x=\"51.378125\" y=\"95.094219\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n","      </g>\n","     </g>\n","     <g id=\"text_8\">\n","      <!-- $\\mathdefault{10^{0}}$ -->\n","      <g transform=\"translate(26.778125 98.893437)scale(0.1 -0.1)\">\n","       <use xlink:href=\"#DejaVuSans-31\" transform=\"translate(0 0.765625)\"/>\n","       <use xlink:href=\"#DejaVuSans-30\" transform=\"translate(63.623047 0.765625)\"/>\n","       <use xlink:href=\"#DejaVuSans-30\" transform=\"translate(128.203125 39.046875)scale(0.7)\"/>\n","      </g>\n","     </g>\n","    </g>\n","    <g id=\"ytick_3\">\n","     <g id=\"line2d_15\">\n","      <path d=\"M 51.378125 59.134313 \n","L 246.678125 59.134313 \n","\" clip-path=\"url(#p97ac790d64)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n","     </g>\n","     <g id=\"line2d_16\">\n","      <g>\n","       <use xlink:href=\"#m5d32c10c90\" x=\"51.378125\" y=\"59.134313\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n","      </g>\n","     </g>\n","     <g id=\"text_9\">\n","      <!-- $\\mathdefault{10^{1}}$ -->\n","      <g transform=\"translate(26.778125 62.933532)scale(0.1 -0.1)\">\n","       <use xlink:href=\"#DejaVuSans-31\" transform=\"translate(0 0.684375)\"/>\n","       <use xlink:href=\"#DejaVuSans-30\" transform=\"translate(63.623047 0.684375)\"/>\n","       <use xlink:href=\"#DejaVuSans-31\" transform=\"translate(128.203125 38.965625)scale(0.7)\"/>\n","      </g>\n","     </g>\n","    </g>\n","    <g id=\"ytick_4\">\n","     <g id=\"line2d_17\">\n","      <path d=\"M 51.378125 23.174408 \n","L 246.678125 23.174408 \n","\" clip-path=\"url(#p97ac790d64)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n","     </g>\n","     <g id=\"line2d_18\">\n","      <g>\n","       <use xlink:href=\"#m5d32c10c90\" x=\"51.378125\" y=\"23.174408\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n","      </g>\n","     </g>\n","     <g id=\"text_10\">\n","      <!-- $\\mathdefault{10^{2}}$ -->\n","      <g transform=\"translate(26.778125 26.973626)scale(0.1 -0.1)\">\n","       <use xlink:href=\"#DejaVuSans-31\" transform=\"translate(0 0.765625)\"/>\n","       <use xlink:href=\"#DejaVuSans-30\" transform=\"translate(63.623047 0.765625)\"/>\n","       <use xlink:href=\"#DejaVuSans-32\" transform=\"translate(128.203125 39.046875)scale(0.7)\"/>\n","      </g>\n","     </g>\n","    </g>\n","    <g id=\"ytick_5\">\n","     <g id=\"line2d_19\">\n","      <defs>\n","       <path id=\"mfa9bd9e3c7\" d=\"M 0 0 \n","L -2 0 \n","\" style=\"stroke: #000000; stroke-width: 0.6\"/>\n","      </defs>\n","      <g>\n","       <use xlink:href=\"#mfa9bd9e3c7\" x=\"51.378125\" y=\"141.879135\" style=\"stroke: #000000; stroke-width: 0.6\"/>\n","      </g>\n","     </g>\n","    </g>\n","    <g id=\"ytick_6\">\n","     <g id=\"line2d_20\">\n","      <g>\n","       <use xlink:href=\"#mfa9bd9e3c7\" x=\"51.378125\" y=\"139.031784\" style=\"stroke: #000000; stroke-width: 0.6\"/>\n","      </g>\n","     </g>\n","    </g>\n","    <g id=\"ytick_7\">\n","     <g id=\"line2d_21\">\n","      <g>\n","       <use xlink:href=\"#mfa9bd9e3c7\" x=\"51.378125\" y=\"136.624384\" style=\"stroke: #000000; stroke-width: 0.6\"/>\n","      </g>\n","     </g>\n","    </g>\n","    <g id=\"ytick_8\">\n","     <g id=\"line2d_22\">\n","      <g>\n","       <use xlink:href=\"#mfa9bd9e3c7\" x=\"51.378125\" y=\"134.538999\" style=\"stroke: #000000; stroke-width: 0.6\"/>\n","      </g>\n","     </g>\n","    </g>\n","    <g id=\"ytick_9\">\n","     <g id=\"line2d_23\">\n","      <g>\n","       <use xlink:href=\"#mfa9bd9e3c7\" x=\"51.378125\" y=\"132.699559\" style=\"stroke: #000000; stroke-width: 0.6\"/>\n","      </g>\n","     </g>\n","    </g>\n","    <g id=\"ytick_10\">\n","     <g id=\"line2d_24\">\n","      <g>\n","       <use xlink:href=\"#mfa9bd9e3c7\" x=\"51.378125\" y=\"120.229114\" style=\"stroke: #000000; stroke-width: 0.6\"/>\n","      </g>\n","     </g>\n","    </g>\n","    <g id=\"ytick_11\">\n","     <g id=\"line2d_25\">\n","      <g>\n","       <use xlink:href=\"#mfa9bd9e3c7\" x=\"51.378125\" y=\"113.896889\" style=\"stroke: #000000; stroke-width: 0.6\"/>\n","      </g>\n","     </g>\n","    </g>\n","    <g id=\"ytick_12\">\n","     <g id=\"line2d_26\">\n","      <g>\n","       <use xlink:href=\"#mfa9bd9e3c7\" x=\"51.378125\" y=\"109.404104\" style=\"stroke: #000000; stroke-width: 0.6\"/>\n","      </g>\n","     </g>\n","    </g>\n","    <g id=\"ytick_13\">\n","     <g id=\"line2d_27\">\n","      <g>\n","       <use xlink:href=\"#mfa9bd9e3c7\" x=\"51.378125\" y=\"105.919229\" style=\"stroke: #000000; stroke-width: 0.6\"/>\n","      </g>\n","     </g>\n","    </g>\n","    <g id=\"ytick_14\">\n","     <g id=\"line2d_28\">\n","      <g>\n","       <use xlink:href=\"#mfa9bd9e3c7\" x=\"51.378125\" y=\"103.071879\" style=\"stroke: #000000; stroke-width: 0.6\"/>\n","      </g>\n","     </g>\n","    </g>\n","    <g id=\"ytick_15\">\n","     <g id=\"line2d_29\">\n","      <g>\n","       <use xlink:href=\"#mfa9bd9e3c7\" x=\"51.378125\" y=\"100.664479\" style=\"stroke: #000000; stroke-width: 0.6\"/>\n","      </g>\n","     </g>\n","    </g>\n","    <g id=\"ytick_16\">\n","     <g id=\"line2d_30\">\n","      <g>\n","       <use xlink:href=\"#mfa9bd9e3c7\" x=\"51.378125\" y=\"98.579094\" style=\"stroke: #000000; stroke-width: 0.6\"/>\n","      </g>\n","     </g>\n","    </g>\n","    <g id=\"ytick_17\">\n","     <g id=\"line2d_31\">\n","      <g>\n","       <use xlink:href=\"#mfa9bd9e3c7\" x=\"51.378125\" y=\"96.739654\" style=\"stroke: #000000; stroke-width: 0.6\"/>\n","      </g>\n","     </g>\n","    </g>\n","    <g id=\"ytick_18\">\n","     <g id=\"line2d_32\">\n","      <g>\n","       <use xlink:href=\"#mfa9bd9e3c7\" x=\"51.378125\" y=\"84.269209\" style=\"stroke: #000000; stroke-width: 0.6\"/>\n","      </g>\n","     </g>\n","    </g>\n","    <g id=\"ytick_19\">\n","     <g id=\"line2d_33\">\n","      <g>\n","       <use xlink:href=\"#mfa9bd9e3c7\" x=\"51.378125\" y=\"77.936983\" style=\"stroke: #000000; stroke-width: 0.6\"/>\n","      </g>\n","     </g>\n","    </g>\n","    <g id=\"ytick_20\">\n","     <g id=\"line2d_34\">\n","      <g>\n","       <use xlink:href=\"#mfa9bd9e3c7\" x=\"51.378125\" y=\"73.444198\" style=\"stroke: #000000; stroke-width: 0.6\"/>\n","      </g>\n","     </g>\n","    </g>\n","    <g id=\"ytick_21\">\n","     <g id=\"line2d_35\">\n","      <g>\n","       <use xlink:href=\"#mfa9bd9e3c7\" x=\"51.378125\" y=\"69.959323\" style=\"stroke: #000000; stroke-width: 0.6\"/>\n","      </g>\n","     </g>\n","    </g>\n","    <g id=\"ytick_22\">\n","     <g id=\"line2d_36\">\n","      <g>\n","       <use xlink:href=\"#mfa9bd9e3c7\" x=\"51.378125\" y=\"67.111973\" style=\"stroke: #000000; stroke-width: 0.6\"/>\n","      </g>\n","     </g>\n","    </g>\n","    <g id=\"ytick_23\">\n","     <g id=\"line2d_37\">\n","      <g>\n","       <use xlink:href=\"#mfa9bd9e3c7\" x=\"51.378125\" y=\"64.704573\" style=\"stroke: #000000; stroke-width: 0.6\"/>\n","      </g>\n","     </g>\n","    </g>\n","    <g id=\"ytick_24\">\n","     <g id=\"line2d_38\">\n","      <g>\n","       <use xlink:href=\"#mfa9bd9e3c7\" x=\"51.378125\" y=\"62.619188\" style=\"stroke: #000000; stroke-width: 0.6\"/>\n","      </g>\n","     </g>\n","    </g>\n","    <g id=\"ytick_25\">\n","     <g id=\"line2d_39\">\n","      <g>\n","       <use xlink:href=\"#mfa9bd9e3c7\" x=\"51.378125\" y=\"60.779748\" style=\"stroke: #000000; stroke-width: 0.6\"/>\n","      </g>\n","     </g>\n","    </g>\n","    <g id=\"ytick_26\">\n","     <g id=\"line2d_40\">\n","      <g>\n","       <use xlink:href=\"#mfa9bd9e3c7\" x=\"51.378125\" y=\"48.309303\" style=\"stroke: #000000; stroke-width: 0.6\"/>\n","      </g>\n","     </g>\n","    </g>\n","    <g id=\"ytick_27\">\n","     <g id=\"line2d_41\">\n","      <g>\n","       <use xlink:href=\"#mfa9bd9e3c7\" x=\"51.378125\" y=\"41.977078\" style=\"stroke: #000000; stroke-width: 0.6\"/>\n","      </g>\n","     </g>\n","    </g>\n","    <g id=\"ytick_28\">\n","     <g id=\"line2d_42\">\n","      <g>\n","       <use xlink:href=\"#mfa9bd9e3c7\" x=\"51.378125\" y=\"37.484293\" style=\"stroke: #000000; stroke-width: 0.6\"/>\n","      </g>\n","     </g>\n","    </g>\n","    <g id=\"ytick_29\">\n","     <g id=\"line2d_43\">\n","      <g>\n","       <use xlink:href=\"#mfa9bd9e3c7\" x=\"51.378125\" y=\"33.999418\" style=\"stroke: #000000; stroke-width: 0.6\"/>\n","      </g>\n","     </g>\n","    </g>\n","    <g id=\"ytick_30\">\n","     <g id=\"line2d_44\">\n","      <g>\n","       <use xlink:href=\"#mfa9bd9e3c7\" x=\"51.378125\" y=\"31.152068\" style=\"stroke: #000000; stroke-width: 0.6\"/>\n","      </g>\n","     </g>\n","    </g>\n","    <g id=\"ytick_31\">\n","     <g id=\"line2d_45\">\n","      <g>\n","       <use xlink:href=\"#mfa9bd9e3c7\" x=\"51.378125\" y=\"28.744667\" style=\"stroke: #000000; stroke-width: 0.6\"/>\n","      </g>\n","     </g>\n","    </g>\n","    <g id=\"ytick_32\">\n","     <g id=\"line2d_46\">\n","      <g>\n","       <use xlink:href=\"#mfa9bd9e3c7\" x=\"51.378125\" y=\"26.659282\" style=\"stroke: #000000; stroke-width: 0.6\"/>\n","      </g>\n","     </g>\n","    </g>\n","    <g id=\"ytick_33\">\n","     <g id=\"line2d_47\">\n","      <g>\n","       <use xlink:href=\"#mfa9bd9e3c7\" x=\"51.378125\" y=\"24.819843\" style=\"stroke: #000000; stroke-width: 0.6\"/>\n","      </g>\n","     </g>\n","    </g>\n","    <g id=\"ytick_34\">\n","     <g id=\"line2d_48\">\n","      <g>\n","       <use xlink:href=\"#mfa9bd9e3c7\" x=\"51.378125\" y=\"12.349397\" style=\"stroke: #000000; stroke-width: 0.6\"/>\n","      </g>\n","     </g>\n","    </g>\n","    <g id=\"text_11\">\n","     <!-- loss -->\n","     <g transform=\"translate(14.798437 84.807812)rotate(-90)scale(0.1 -0.1)\">\n","      <defs>\n","       <path id=\"DejaVuSans-6c\" d=\"M 603 4863 \n","L 1178 4863 \n","L 1178 0 \n","L 603 0 \n","L 603 4863 \n","z\n","\" transform=\"scale(0.015625)\"/>\n","      </defs>\n","      <use xlink:href=\"#DejaVuSans-6c\"/>\n","      <use xlink:href=\"#DejaVuSans-6f\" x=\"27.783203\"/>\n","      <use xlink:href=\"#DejaVuSans-73\" x=\"88.964844\"/>\n","      <use xlink:href=\"#DejaVuSans-73\" x=\"141.064453\"/>\n","     </g>\n","    </g>\n","   </g>\n","   <g id=\"line2d_49\">\n","    <path d=\"M 51.378125 66.609229 \n","L 61.657072 102.92158 \n","L 71.93602 124.166574 \n","L 82.214967 129.389785 \n","L 92.493914 130.171889 \n","L 102.772862 130.403184 \n","L 113.051809 130.637817 \n","L 123.330757 131.091839 \n","L 133.609704 131.591448 \n","L 143.888651 131.989853 \n","L 154.167599 132.540155 \n","L 164.446546 133.046585 \n","L 174.725493 133.574638 \n","L 185.004441 134.073878 \n","L 195.283388 134.510888 \n","L 205.562336 135.01178 \n","L 215.841283 135.521189 \n","L 226.12023 135.992137 \n","L 236.399178 136.381117 \n","L 246.678125 136.922727 \n","\" clip-path=\"url(#p97ac790d64)\" style=\"fill: none; stroke: #1f77b4; stroke-width: 1.5; stroke-linecap: square\"/>\n","   </g>\n","   <g id=\"line2d_50\">\n","    <path d=\"M 51.378125 13.377273 \n","L 61.657072 19.102244 \n","L 71.93602 24.613319 \n","L 82.214967 29.953459 \n","L 92.493914 35.111297 \n","L 102.772862 40.045542 \n","L 113.051809 44.698434 \n","L 123.330757 49.00995 \n","L 133.609704 52.909019 \n","L 143.888651 56.354886 \n","L 154.167599 59.31546 \n","L 164.446546 61.804721 \n","L 174.725493 63.873401 \n","L 185.004441 65.541152 \n","L 195.283388 66.885531 \n","L 205.562336 67.993916 \n","L 215.841283 68.902959 \n","L 226.12023 69.678381 \n","L 236.399178 70.352107 \n","L 246.678125 70.951613 \n","\" clip-path=\"url(#p97ac790d64)\" style=\"fill: none; stroke-dasharray: 5.55,2.4; stroke-dashoffset: 0; stroke: #bf00bf; stroke-width: 1.5\"/>\n","   </g>\n","   <g id=\"patch_3\">\n","    <path d=\"M 51.378125 143.1 \n","L 51.378125 7.2 \n","\" style=\"fill: none; stroke: #000000; stroke-width: 0.8; stroke-linejoin: miter; stroke-linecap: square\"/>\n","   </g>\n","   <g id=\"patch_4\">\n","    <path d=\"M 246.678125 143.1 \n","L 246.678125 7.2 \n","\" style=\"fill: none; stroke: #000000; stroke-width: 0.8; stroke-linejoin: miter; stroke-linecap: square\"/>\n","   </g>\n","   <g id=\"patch_5\">\n","    <path d=\"M 51.378125 143.1 \n","L 246.678125 143.1 \n","\" style=\"fill: none; stroke: #000000; stroke-width: 0.8; stroke-linejoin: miter; stroke-linecap: square\"/>\n","   </g>\n","   <g id=\"patch_6\">\n","    <path d=\"M 51.378125 7.2 \n","L 246.678125 7.2 \n","\" style=\"fill: none; stroke: #000000; stroke-width: 0.8; stroke-linejoin: miter; stroke-linecap: square\"/>\n","   </g>\n","   <g id=\"legend_1\">\n","    <g id=\"patch_7\">\n","     <path d=\"M 184.403125 44.55625 \n","L 239.678125 44.55625 \n","Q 241.678125 44.55625 241.678125 42.55625 \n","L 241.678125 14.2 \n","Q 241.678125 12.2 239.678125 12.2 \n","L 184.403125 12.2 \n","Q 182.403125 12.2 182.403125 14.2 \n","L 182.403125 42.55625 \n","Q 182.403125 44.55625 184.403125 44.55625 \n","z\n","\" style=\"fill: #ffffff; opacity: 0.8; stroke: #cccccc; stroke-linejoin: miter\"/>\n","    </g>\n","    <g id=\"line2d_51\">\n","     <path d=\"M 186.403125 20.298437 \n","L 196.403125 20.298437 \n","L 206.403125 20.298437 \n","\" style=\"fill: none; stroke: #1f77b4; stroke-width: 1.5; stroke-linecap: square\"/>\n","    </g>\n","    <g id=\"text_12\">\n","     <!-- train -->\n","     <g transform=\"translate(214.403125 23.798437)scale(0.1 -0.1)\">\n","      <defs>\n","       <path id=\"DejaVuSans-74\" d=\"M 1172 4494 \n","L 1172 3500 \n","L 2356 3500 \n","L 2356 3053 \n","L 1172 3053 \n","L 1172 1153 \n","Q 1172 725 1289 603 \n","Q 1406 481 1766 481 \n","L 2356 481 \n","L 2356 0 \n","L 1766 0 \n","Q 1100 0 847 248 \n","Q 594 497 594 1153 \n","L 594 3053 \n","L 172 3053 \n","L 172 3500 \n","L 594 3500 \n","L 594 4494 \n","L 1172 4494 \n","z\n","\" transform=\"scale(0.015625)\"/>\n","       <path id=\"DejaVuSans-72\" d=\"M 2631 2963 \n","Q 2534 3019 2420 3045 \n","Q 2306 3072 2169 3072 \n","Q 1681 3072 1420 2755 \n","Q 1159 2438 1159 1844 \n","L 1159 0 \n","L 581 0 \n","L 581 3500 \n","L 1159 3500 \n","L 1159 2956 \n","Q 1341 3275 1631 3429 \n","Q 1922 3584 2338 3584 \n","Q 2397 3584 2469 3576 \n","Q 2541 3569 2628 3553 \n","L 2631 2963 \n","z\n","\" transform=\"scale(0.015625)\"/>\n","       <path id=\"DejaVuSans-61\" d=\"M 2194 1759 \n","Q 1497 1759 1228 1600 \n","Q 959 1441 959 1056 \n","Q 959 750 1161 570 \n","Q 1363 391 1709 391 \n","Q 2188 391 2477 730 \n","Q 2766 1069 2766 1631 \n","L 2766 1759 \n","L 2194 1759 \n","z\n","M 3341 1997 \n","L 3341 0 \n","L 2766 0 \n","L 2766 531 \n","Q 2569 213 2275 61 \n","Q 1981 -91 1556 -91 \n","Q 1019 -91 701 211 \n","Q 384 513 384 1019 \n","Q 384 1609 779 1909 \n","Q 1175 2209 1959 2209 \n","L 2766 2209 \n","L 2766 2266 \n","Q 2766 2663 2505 2880 \n","Q 2244 3097 1772 3097 \n","Q 1472 3097 1187 3025 \n","Q 903 2953 641 2809 \n","L 641 3341 \n","Q 956 3463 1253 3523 \n","Q 1550 3584 1831 3584 \n","Q 2591 3584 2966 3190 \n","Q 3341 2797 3341 1997 \n","z\n","\" transform=\"scale(0.015625)\"/>\n","       <path id=\"DejaVuSans-69\" d=\"M 603 3500 \n","L 1178 3500 \n","L 1178 0 \n","L 603 0 \n","L 603 3500 \n","z\n","M 603 4863 \n","L 1178 4863 \n","L 1178 4134 \n","L 603 4134 \n","L 603 4863 \n","z\n","\" transform=\"scale(0.015625)\"/>\n","       <path id=\"DejaVuSans-6e\" d=\"M 3513 2113 \n","L 3513 0 \n","L 2938 0 \n","L 2938 2094 \n","Q 2938 2591 2744 2837 \n","Q 2550 3084 2163 3084 \n","Q 1697 3084 1428 2787 \n","Q 1159 2491 1159 1978 \n","L 1159 0 \n","L 581 0 \n","L 581 3500 \n","L 1159 3500 \n","L 1159 2956 \n","Q 1366 3272 1645 3428 \n","Q 1925 3584 2291 3584 \n","Q 2894 3584 3203 3211 \n","Q 3513 2838 3513 2113 \n","z\n","\" transform=\"scale(0.015625)\"/>\n","      </defs>\n","      <use xlink:href=\"#DejaVuSans-74\"/>\n","      <use xlink:href=\"#DejaVuSans-72\" x=\"39.208984\"/>\n","      <use xlink:href=\"#DejaVuSans-61\" x=\"80.322266\"/>\n","      <use xlink:href=\"#DejaVuSans-69\" x=\"141.601562\"/>\n","      <use xlink:href=\"#DejaVuSans-6e\" x=\"169.384766\"/>\n","     </g>\n","    </g>\n","    <g id=\"line2d_52\">\n","     <path d=\"M 186.403125 34.976562 \n","L 196.403125 34.976562 \n","L 206.403125 34.976562 \n","\" style=\"fill: none; stroke-dasharray: 5.55,2.4; stroke-dashoffset: 0; stroke: #bf00bf; stroke-width: 1.5\"/>\n","    </g>\n","    <g id=\"text_13\">\n","     <!-- test -->\n","     <g transform=\"translate(214.403125 38.476562)scale(0.1 -0.1)\">\n","      <use xlink:href=\"#DejaVuSans-74\"/>\n","      <use xlink:href=\"#DejaVuSans-65\" x=\"39.208984\"/>\n","      <use xlink:href=\"#DejaVuSans-73\" x=\"100.732422\"/>\n","      <use xlink:href=\"#DejaVuSans-74\" x=\"152.832031\"/>\n","     </g>\n","    </g>\n","   </g>\n","  </g>\n"," </g>\n"," <defs>\n","  <clipPath id=\"p97ac790d64\">\n","   <rect x=\"51.378125\" y=\"7.2\" width=\"195.3\" height=\"135.9\"/>\n","  </clipPath>\n"," </defs>\n","</svg>\n"],"text/plain":["<Figure size 252x180 with 1 Axes>"]},"metadata":{"needs_background":"light"},"output_type":"display_data"}],"source":["train_concise(3)"],"id":"30767598"},{"cell_type":"markdown","metadata":{"origin_pos":32,"id":"e6821ce9"},"source":["到目前为止，我们只接触到一个简单线性函数的概念。\n","此外，由什么构成一个简单的非线性函数可能是一个更复杂的问题。\n","例如，[再生核希尔伯特空间（RKHS）](https://en.wikipedia.org/wiki/Reproducing_kernel_Hilbert_space)\n","允许在非线性环境中应用为线性函数引入的工具。\n","不幸的是，基于RKHS的算法往往难以应用到大型、高维的数据。\n","在这本书中，我们将默认使用简单的启发式方法，即在深层网络的所有层上应用权重衰减。\n","\n","## 小结\n","\n","* 正则化是处理过拟合的常用方法：在训练集的损失函数中加入惩罚项，以降低学习到的模型的复杂度。\n","* 保持模型简单的一个特别的选择是使用$L_2$惩罚的权重衰减。这会导致学习算法更新步骤中的权重衰减。\n","* 权重衰减功能在深度学习框架的优化器中提供。\n","* 在同一训练代码实现中，不同的参数集可以有不同的更新行为。\n","\n","## 练习\n","\n","1. 在本节的估计问题中使用$\\lambda$的值进行实验。绘制训练和测试精度关于$\\lambda$的函数。你观察到了什么？\n","1. 使用验证集来找到最佳值$\\lambda$。它真的是最优值吗？这有关系吗？\n","1. 如果我们使用$\\sum_i |w_i|$作为我们选择的惩罚（$L_1$正则化），那么更新方程会是什么样子？\n","1. 我们知道$\\|\\mathbf{w}\\|^2 = \\mathbf{w}^\\top \\mathbf{w}$。你能找到类似的矩阵方程吗（见 :numref:`subsec_lin-algebra-norms` 中的Frobenius范数）？\n","1. 回顾训练误差和泛化误差之间的关系。除了权重衰减、增加训练数据、使用适当复杂度的模型之外，你还能想出其他什么方法来处理过拟合？\n","1. 在贝叶斯统计中，我们使用先验和似然的乘积，通过公式$P(w \\mid x) \\propto P(x \\mid w) P(w)$得到后验。如何得到带正则化的$P(w)$？\n"],"id":"e6821ce9"},{"cell_type":"markdown","metadata":{"origin_pos":34,"tab":["pytorch"],"id":"f16cd59b"},"source":["[Discussions](https://discuss.d2l.ai/t/1808)\n"],"id":"f16cd59b"},{"cell_type":"markdown","metadata":{"origin_pos":0,"id":"gX6e_bMFhB9B"},"source":["# 权重衰减\n",":label:`sec_weight_decay`\n","\n","前一节我们描述了过拟合的问题，本节我们将介绍一些正则化模型的技术。\n","我们总是可以通过去收集更多的训练数据来缓解过拟合。\n","但这可能成本很高，耗时颇多，或者完全超出我们的控制，因而在短期内不可能做到。\n","假设我们已经拥有尽可能多的高质量数据，我们便可以将重点放在正则化技术上。\n","\n","回想一下，在多项式回归的例子（ :numref:`sec_model_selection`）中，\n","我们可以通过调整拟合多项式的阶数来限制模型的容量。\n","实际上，限制特征的数量是缓解过拟合的一种常用技术。\n","然而，简单地丢弃特征对于这项工作来说可能过于生硬。\n","我们继续思考多项式回归的例子，考虑高维输入可能发生的情况。\n","多项式对多变量数据的自然扩展称为*单项式*（monomials），\n","也可以说是变量幂的乘积。\n","单项式的阶数是幂的和。\n","例如，$x_1^2 x_2$和$x_3 x_5^2$都是3次单项式。\n","\n","注意，随着阶数$d$的增长，带有阶数$d$的项数迅速增加。 \n","给定$k$个变量，阶数为$d$的项的个数为\n","${k - 1 + d} \\choose {k - 1}$，即$C^{k-1}_{k-1+d} = \\frac{(k-1+d)!}{(d)!(k-1)!}$。\n","因此即使是阶数上的微小变化，比如从$2$到$3$，也会显著增加我们模型的复杂性。\n","仅仅通过简单的限制特征数量（在多项式回归中体现为限制阶数），可能仍然使模型在过简单和过复杂中徘徊，\n","我们需要一个更细粒度的工具来调整函数的复杂性，使其达到一个合适的平衡位置。\n","## 范数与权重衰减\n","\n","在 :numref:`subsec_lin-algebra-norms`中，\n","我们已经描述了$L_2$范数和$L_1$范数，\n","它们是更为一般的$L_p$范数的特殊情况。\n","(~~权重衰减是最广泛使用的正则化的技术之一~~)\n","在训练参数化机器学习模型时，\n","*权重衰减*（weight decay）是最广泛使用的正则化的技术之一，\n","它通常也被称为$L_2$*正则化*。\n","这项技术通过函数与零的距离来衡量函数的复杂度，\n","因为在所有函数$f$中，函数$f = 0$（所有输入都得到值$0$）\n","在某种意义上是最简单的。\n","但是我们应该如何精确地测量一个函数和零之间的距离呢？\n","没有一个正确的答案。\n","事实上，函数分析和巴拿赫空间理论的研究，都在致力于回答这个问题。\n","\n","一种简单的方法是通过线性函数\n","$f(\\mathbf{x}) = \\mathbf{w}^\\top \\mathbf{x}$\n","中的权重向量的某个范数来度量其复杂性，\n","例如$\\| \\mathbf{w} \\|^2$。\n","要保证权重向量比较小，\n","最常用方法是将其范数作为惩罚项加到最小化损失的问题中。\n","将原来的训练目标*最小化训练标签上的预测损失*，\n","调整为*最小化预测损失和惩罚项之和*。\n","现在，如果我们的权重向量增长的太大，\n","我们的学习算法可能会更集中于最小化权重范数$\\| \\mathbf{w} \\|^2$。\n","这正是我们想要的。\n","让我们回顾一下 :numref:`sec_linear_regression`中的线性回归例子。\n","我们的损失由下式给出：\n","\n","$$L(\\mathbf{w}, b) = \\frac{1}{n}\\sum_{i=1}^n \\frac{1}{2}\\left(\\mathbf{w}^\\top \\mathbf{x}^{(i)} + b - y^{(i)}\\right)^2.$$\n","\n","回想一下，$\\mathbf{x}^{(i)}$是样本$i$的特征，\n","$y^{(i)}$是样本$i$的标签，\n","$(\\mathbf{w}, b)$是权重和偏置参数。\n","为了惩罚权重向量的大小，\n","我们必须以某种方式在损失函数中添加$\\| \\mathbf{w} \\|^2$，\n","但是模型应该如何平衡这个新的额外惩罚的损失？\n","实际上，我们通过*正则化常数*$\\lambda$来描述这种权衡，\n","这是一个非负超参数，我们使用验证数据拟合：\n","\n","$$L(\\mathbf{w}, b) + \\frac{\\lambda}{2} \\|\\mathbf{w}\\|^2,$$\n","\n","对于$\\lambda = 0$，我们恢复了原来的损失函数。\n","对于$\\lambda > 0$，我们限制$\\| \\mathbf{w} \\|$的大小。\n","这里我们仍然除以$2$：当我们取一个二次函数的导数时，\n","$2$和$1/2$会抵消，以确保更新表达式看起来既漂亮又简单。\n","你可能会想知道为什么我们使用平方范数而不是标准范数（即欧几里得距离）？\n","我们这样做是为了便于计算。\n","通过平方$L_2$范数，我们去掉平方根，留下权重向量每个分量的平方和。\n","这使得惩罚的导数很容易计算：导数的和等于和的导数。\n","\n","此外，你可能会问为什么我们首先使用$L_2$范数，而不是$L_1$范数。\n","事实上，这个选择在整个统计领域中都是有效的和受欢迎的。\n","$L_2$正则化线性模型构成经典的*岭回归*（ridge regression）算法，\n","$L_1$正则化线性回归是统计学中类似的基本模型，\n","通常被称为*套索回归*（lasso regression）。\n","使用$L_2$范数的一个原因是它对权重向量的大分量施加了巨大的惩罚。\n","这使得我们的学习算法偏向于在大量特征上均匀分布权重的模型。\n","在实践中，这可能使它们对单个变量中的观测误差更为稳定。\n","相比之下，$L_1$惩罚会导致模型将权重集中在一小部分特征上，\n","而将其他权重清除为零。\n","这称为*特征选择*（feature selection），这可能是其他场景下需要的。\n","\n","使用与 :eqref:`eq_linreg_batch_update`中的相同符号，\n","$L_2$正则化回归的小批量随机梯度下降更新如下式：\n","\n","$$\n","\\begin{aligned}\n","\\mathbf{w} & \\leftarrow \\left(1- \\eta\\lambda \\right) \\mathbf{w} - \\frac{\\eta}{|\\mathcal{B}|} \\sum_{i \\in \\mathcal{B}} \\mathbf{x}^{(i)} \\left(\\mathbf{w}^\\top \\mathbf{x}^{(i)} + b - y^{(i)}\\right).\n","\\end{aligned}\n","$$\n","\n","根据之前章节所讲的，我们根据估计值与观测值之间的差异来更新$\\mathbf{w}$。\n","然而，我们同时也在试图将$\\mathbf{w}$的大小缩小到零。\n","这就是为什么这种方法有时被称为*权重衰减*。\n","我们仅考虑惩罚项，优化算法在训练的每一步*衰减*权重。\n","与特征选择相比，权重衰减为我们提供了一种连续的机制来调整函数的复杂度。\n","较小的$\\lambda$值对应较少约束的$\\mathbf{w}$，\n","而较大的$\\lambda$值对$\\mathbf{w}$的约束更大。\n","\n","是否对相应的偏置$b^2$进行惩罚在不同的实践中会有所不同，\n","在神经网络的不同层中也会有所不同。\n","通常，网络输出层的偏置项不会被正则化。\n","\n","## 高维线性回归\n","\n","我们通过一个简单的例子来演示权重衰减。\n"],"id":"gX6e_bMFhB9B"},{"cell_type":"code","execution_count":null,"metadata":{"execution":{"iopub.execute_input":"2022-07-31T02:22:47.330465Z","iopub.status.busy":"2022-07-31T02:22:47.329932Z","iopub.status.idle":"2022-07-31T02:22:49.309063Z","shell.execute_reply":"2022-07-31T02:22:49.308346Z"},"origin_pos":2,"tab":["pytorch"],"id":"cdY6ndrghB9C"},"outputs":[],"source":["%matplotlib inline\n","import torch\n","from torch import nn\n","from d2l import torch as d2l"],"id":"cdY6ndrghB9C"},{"cell_type":"markdown","metadata":{"origin_pos":4,"id":"RvNKETvshB9C"},"source":["首先，我们[**像以前一样生成一些数据**]，生成公式如下：\n","\n","(**$$y = 0.05 + \\sum_{i = 1}^d 0.01 x_i + \\epsilon \\text{ where }\n","\\epsilon \\sim \\mathcal{N}(0, 0.01^2).$$**)\n","\n","我们选择标签是关于输入的线性函数。\n","标签同时被均值为0，标准差为0.01高斯噪声破坏。\n","为了使过拟合的效果更加明显，我们可以将问题的维数增加到$d = 200$，\n","并使用一个只包含20个样本的小训练集。\n"],"id":"RvNKETvshB9C"},{"cell_type":"code","execution_count":null,"metadata":{"execution":{"iopub.execute_input":"2022-07-31T02:22:49.312954Z","iopub.status.busy":"2022-07-31T02:22:49.312444Z","iopub.status.idle":"2022-07-31T02:22:49.319046Z","shell.execute_reply":"2022-07-31T02:22:49.318390Z"},"origin_pos":5,"tab":["pytorch"],"id":"j5JnmmnxhB9C"},"outputs":[],"source":["n_train, n_test, num_inputs, batch_size = 20, 100, 200, 5\n","true_w, true_b = torch.ones((num_inputs, 1)) * 0.01, 0.05\n","train_data = d2l.synthetic_data(true_w, true_b, n_train)\n","train_iter = d2l.load_array(train_data, batch_size)\n","test_data = d2l.synthetic_data(true_w, true_b, n_test)\n","test_iter = d2l.load_array(test_data, batch_size, is_train=False)"],"id":"j5JnmmnxhB9C"},{"cell_type":"markdown","metadata":{"origin_pos":6,"id":"6C1vT0VuhB9C"},"source":["## 从零开始实现\n","\n","下面我们将从头开始实现权重衰减，只需将$L_2$的平方惩罚添加到原始目标函数中。\n","\n","### [**初始化模型参数**]\n","\n","首先，我们将定义一个函数来随机初始化模型参数。\n"],"id":"6C1vT0VuhB9C"},{"cell_type":"code","execution_count":null,"metadata":{"execution":{"iopub.execute_input":"2022-07-31T02:22:49.323345Z","iopub.status.busy":"2022-07-31T02:22:49.322872Z","iopub.status.idle":"2022-07-31T02:22:49.327215Z","shell.execute_reply":"2022-07-31T02:22:49.326404Z"},"origin_pos":8,"tab":["pytorch"],"id":"WoGzIelThB9C"},"outputs":[],"source":["def init_params():\n","    w = torch.normal(0, 1, size=(num_inputs, 1), requires_grad=True)\n","    b = torch.zeros(1, requires_grad=True)\n","    return [w, b]"],"id":"WoGzIelThB9C"},{"cell_type":"markdown","metadata":{"origin_pos":10,"id":"xUR7fn_QhB9D"},"source":["### (**定义$L_2$范数惩罚**)\n","\n","实现这一惩罚最方便的方法是对所有项求平方后并将它们求和。\n"],"id":"xUR7fn_QhB9D"},{"cell_type":"code","execution_count":null,"metadata":{"execution":{"iopub.execute_input":"2022-07-31T02:22:49.331227Z","iopub.status.busy":"2022-07-31T02:22:49.330783Z","iopub.status.idle":"2022-07-31T02:22:49.334599Z","shell.execute_reply":"2022-07-31T02:22:49.333840Z"},"origin_pos":12,"tab":["pytorch"],"id":"CJRucJZ9hB9D"},"outputs":[],"source":["def l2_penalty(w):\n","    return torch.sum(w.pow(2)) / 2"],"id":"CJRucJZ9hB9D"},{"cell_type":"markdown","metadata":{"origin_pos":14,"id":"0WbU6u84hB9D"},"source":["### [**定义训练代码实现**]\n","\n","下面的代码将模型拟合训练数据集，并在测试数据集上进行评估。\n","从 :numref:`chap_linear`以来，线性网络和平方损失没有变化，\n","所以我们通过`d2l.linreg`和`d2l.squared_loss`导入它们。\n","唯一的变化是损失现在包括了惩罚项。\n"],"id":"0WbU6u84hB9D"},{"cell_type":"code","execution_count":null,"metadata":{"execution":{"iopub.execute_input":"2022-07-31T02:22:49.338662Z","iopub.status.busy":"2022-07-31T02:22:49.338197Z","iopub.status.idle":"2022-07-31T02:22:49.344902Z","shell.execute_reply":"2022-07-31T02:22:49.344113Z"},"origin_pos":16,"tab":["pytorch"],"id":"1RDaXlfGhB9D"},"outputs":[],"source":["def train(lambd):\n","    w, b = init_params()\n","    net, loss = lambda X: d2l.linreg(X, w, b), d2l.squared_loss\n","    num_epochs, lr = 100, 0.003\n","    animator = d2l.Animator(xlabel='epochs', ylabel='loss', yscale='log',\n","                            xlim=[5, num_epochs], legend=['train', 'test'])\n","    for epoch in range(num_epochs):\n","        for X, y in train_iter:\n","            # 增加了L2范数惩罚项，\n","            # 广播机制使l2_penalty(w)成为一个长度为batch_size的向量\n","            l = loss(net(X), y) + lambd * l2_penalty(w)\n","            l.sum().backward()\n","            d2l.sgd([w, b], lr, batch_size)\n","        if (epoch + 1) % 5 == 0:\n","            animator.add(epoch + 1, (d2l.evaluate_loss(net, train_iter, loss),\n","                                     d2l.evaluate_loss(net, test_iter, loss)))\n","    print('w的L2范数是：', torch.norm(w).item())"],"id":"1RDaXlfGhB9D"},{"cell_type":"markdown","metadata":{"origin_pos":18,"id":"LRSJPEYkhB9D"},"source":["### [**忽略正则化直接训练**]\n","\n","我们现在用`lambd = 0`禁用权重衰减后运行这个代码。\n","注意，这里训练误差有了减少，但测试误差没有减少，\n","这意味着出现了严重的过拟合。\n"],"id":"LRSJPEYkhB9D"},{"cell_type":"code","execution_count":null,"metadata":{"execution":{"iopub.execute_input":"2022-07-31T02:22:49.348006Z","iopub.status.busy":"2022-07-31T02:22:49.347416Z","iopub.status.idle":"2022-07-31T02:22:55.832837Z","shell.execute_reply":"2022-07-31T02:22:55.832068Z"},"origin_pos":19,"tab":["pytorch"],"outputId":"7d0c9c39-cae7-4882-8415-f1c012e36c27","id":"jYYnQcyphB9D"},"outputs":[{"name":"stdout","output_type":"stream","text":["w的L2范数是： 12.791678428649902\n"]},{"data":{"image/svg+xml":["<?xml version=\"1.0\" encoding=\"utf-8\" standalone=\"no\"?>\n","<!DOCTYPE svg PUBLIC \"-//W3C//DTD SVG 1.1//EN\"\n","  \"http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd\">\n","<svg xmlns:xlink=\"http://www.w3.org/1999/xlink\" width=\"263.421875pt\" height=\"180.65625pt\" viewBox=\"0 0 263.421875 180.65625\" xmlns=\"http://www.w3.org/2000/svg\" version=\"1.1\">\n"," <metadata>\n","  <rdf:RDF xmlns:dc=\"http://purl.org/dc/elements/1.1/\" xmlns:cc=\"http://creativecommons.org/ns#\" xmlns:rdf=\"http://www.w3.org/1999/02/22-rdf-syntax-ns#\">\n","   <cc:Work>\n","    <dc:type rdf:resource=\"http://purl.org/dc/dcmitype/StillImage\"/>\n","    <dc:date>2022-07-31T02:22:55.759832</dc:date>\n","    <dc:format>image/svg+xml</dc:format>\n","    <dc:creator>\n","     <cc:Agent>\n","      <dc:title>Matplotlib v3.5.1, https://matplotlib.org/</dc:title>\n","     </cc:Agent>\n","    </dc:creator>\n","   </cc:Work>\n","  </rdf:RDF>\n"," </metadata>\n"," <defs>\n","  <style type=\"text/css\">*{stroke-linejoin: round; stroke-linecap: butt}</style>\n"," </defs>\n"," <g id=\"figure_1\">\n","  <g id=\"patch_1\">\n","   <path d=\"M 0 180.65625 \n","L 263.421875 180.65625 \n","L 263.421875 0 \n","L 0 0 \n","L 0 180.65625 \n","z\n","\" style=\"fill: none\"/>\n","  </g>\n","  <g id=\"axes_1\">\n","   <g id=\"patch_2\">\n","    <path d=\"M 51.378125 143.1 \n","L 246.678125 143.1 \n","L 246.678125 7.2 \n","L 51.378125 7.2 \n","z\n","\" style=\"fill: #ffffff\"/>\n","   </g>\n","   <g id=\"matplotlib.axis_1\">\n","    <g id=\"xtick_1\">\n","     <g id=\"line2d_1\">\n","      <path d=\"M 82.214967 143.1 \n","L 82.214967 7.2 \n","\" clip-path=\"url(#paf1508e7f6)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n","     </g>\n","     <g id=\"line2d_2\">\n","      <defs>\n","       <path id=\"m3a97404132\" d=\"M 0 0 \n","L 0 3.5 \n","\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n","      </defs>\n","      <g>\n","       <use xlink:href=\"#m3a97404132\" x=\"82.214967\" y=\"143.1\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n","      </g>\n","     </g>\n","     <g id=\"text_1\">\n","      <!-- 20 -->\n","      <g transform=\"translate(75.852467 157.698438)scale(0.1 -0.1)\">\n","       <defs>\n","        <path id=\"DejaVuSans-32\" d=\"M 1228 531 \n","L 3431 531 \n","L 3431 0 \n","L 469 0 \n","L 469 531 \n","Q 828 903 1448 1529 \n","Q 2069 2156 2228 2338 \n","Q 2531 2678 2651 2914 \n","Q 2772 3150 2772 3378 \n","Q 2772 3750 2511 3984 \n","Q 2250 4219 1831 4219 \n","Q 1534 4219 1204 4116 \n","Q 875 4013 500 3803 \n","L 500 4441 \n","Q 881 4594 1212 4672 \n","Q 1544 4750 1819 4750 \n","Q 2544 4750 2975 4387 \n","Q 3406 4025 3406 3419 \n","Q 3406 3131 3298 2873 \n","Q 3191 2616 2906 2266 \n","Q 2828 2175 2409 1742 \n","Q 1991 1309 1228 531 \n","z\n","\" transform=\"scale(0.015625)\"/>\n","        <path id=\"DejaVuSans-30\" d=\"M 2034 4250 \n","Q 1547 4250 1301 3770 \n","Q 1056 3291 1056 2328 \n","Q 1056 1369 1301 889 \n","Q 1547 409 2034 409 \n","Q 2525 409 2770 889 \n","Q 3016 1369 3016 2328 \n","Q 3016 3291 2770 3770 \n","Q 2525 4250 2034 4250 \n","z\n","M 2034 4750 \n","Q 2819 4750 3233 4129 \n","Q 3647 3509 3647 2328 \n","Q 3647 1150 3233 529 \n","Q 2819 -91 2034 -91 \n","Q 1250 -91 836 529 \n","Q 422 1150 422 2328 \n","Q 422 3509 836 4129 \n","Q 1250 4750 2034 4750 \n","z\n","\" transform=\"scale(0.015625)\"/>\n","       </defs>\n","       <use xlink:href=\"#DejaVuSans-32\"/>\n","       <use xlink:href=\"#DejaVuSans-30\" x=\"63.623047\"/>\n","      </g>\n","     </g>\n","    </g>\n","    <g id=\"xtick_2\">\n","     <g id=\"line2d_3\">\n","      <path d=\"M 123.330757 143.1 \n","L 123.330757 7.2 \n","\" clip-path=\"url(#paf1508e7f6)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n","     </g>\n","     <g id=\"line2d_4\">\n","      <g>\n","       <use xlink:href=\"#m3a97404132\" x=\"123.330757\" y=\"143.1\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n","      </g>\n","     </g>\n","     <g id=\"text_2\">\n","      <!-- 40 -->\n","      <g transform=\"translate(116.968257 157.698438)scale(0.1 -0.1)\">\n","       <defs>\n","        <path id=\"DejaVuSans-34\" d=\"M 2419 4116 \n","L 825 1625 \n","L 2419 1625 \n","L 2419 4116 \n","z\n","M 2253 4666 \n","L 3047 4666 \n","L 3047 1625 \n","L 3713 1625 \n","L 3713 1100 \n","L 3047 1100 \n","L 3047 0 \n","L 2419 0 \n","L 2419 1100 \n","L 313 1100 \n","L 313 1709 \n","L 2253 4666 \n","z\n","\" transform=\"scale(0.015625)\"/>\n","       </defs>\n","       <use xlink:href=\"#DejaVuSans-34\"/>\n","       <use xlink:href=\"#DejaVuSans-30\" x=\"63.623047\"/>\n","      </g>\n","     </g>\n","    </g>\n","    <g id=\"xtick_3\">\n","     <g id=\"line2d_5\">\n","      <path d=\"M 164.446546 143.1 \n","L 164.446546 7.2 \n","\" clip-path=\"url(#paf1508e7f6)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n","     </g>\n","     <g id=\"line2d_6\">\n","      <g>\n","       <use xlink:href=\"#m3a97404132\" x=\"164.446546\" y=\"143.1\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n","      </g>\n","     </g>\n","     <g id=\"text_3\">\n","      <!-- 60 -->\n","      <g transform=\"translate(158.084046 157.698438)scale(0.1 -0.1)\">\n","       <defs>\n","        <path id=\"DejaVuSans-36\" d=\"M 2113 2584 \n","Q 1688 2584 1439 2293 \n","Q 1191 2003 1191 1497 \n","Q 1191 994 1439 701 \n","Q 1688 409 2113 409 \n","Q 2538 409 2786 701 \n","Q 3034 994 3034 1497 \n","Q 3034 2003 2786 2293 \n","Q 2538 2584 2113 2584 \n","z\n","M 3366 4563 \n","L 3366 3988 \n","Q 3128 4100 2886 4159 \n","Q 2644 4219 2406 4219 \n","Q 1781 4219 1451 3797 \n","Q 1122 3375 1075 2522 \n","Q 1259 2794 1537 2939 \n","Q 1816 3084 2150 3084 \n","Q 2853 3084 3261 2657 \n","Q 3669 2231 3669 1497 \n","Q 3669 778 3244 343 \n","Q 2819 -91 2113 -91 \n","Q 1303 -91 875 529 \n","Q 447 1150 447 2328 \n","Q 447 3434 972 4092 \n","Q 1497 4750 2381 4750 \n","Q 2619 4750 2861 4703 \n","Q 3103 4656 3366 4563 \n","z\n","\" transform=\"scale(0.015625)\"/>\n","       </defs>\n","       <use xlink:href=\"#DejaVuSans-36\"/>\n","       <use xlink:href=\"#DejaVuSans-30\" x=\"63.623047\"/>\n","      </g>\n","     </g>\n","    </g>\n","    <g id=\"xtick_4\">\n","     <g id=\"line2d_7\">\n","      <path d=\"M 205.562336 143.1 \n","L 205.562336 7.2 \n","\" clip-path=\"url(#paf1508e7f6)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n","     </g>\n","     <g id=\"line2d_8\">\n","      <g>\n","       <use xlink:href=\"#m3a97404132\" x=\"205.562336\" y=\"143.1\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n","      </g>\n","     </g>\n","     <g id=\"text_4\">\n","      <!-- 80 -->\n","      <g transform=\"translate(199.199836 157.698438)scale(0.1 -0.1)\">\n","       <defs>\n","        <path id=\"DejaVuSans-38\" d=\"M 2034 2216 \n","Q 1584 2216 1326 1975 \n","Q 1069 1734 1069 1313 \n","Q 1069 891 1326 650 \n","Q 1584 409 2034 409 \n","Q 2484 409 2743 651 \n","Q 3003 894 3003 1313 \n","Q 3003 1734 2745 1975 \n","Q 2488 2216 2034 2216 \n","z\n","M 1403 2484 \n","Q 997 2584 770 2862 \n","Q 544 3141 544 3541 \n","Q 544 4100 942 4425 \n","Q 1341 4750 2034 4750 \n","Q 2731 4750 3128 4425 \n","Q 3525 4100 3525 3541 \n","Q 3525 3141 3298 2862 \n","Q 3072 2584 2669 2484 \n","Q 3125 2378 3379 2068 \n","Q 3634 1759 3634 1313 \n","Q 3634 634 3220 271 \n","Q 2806 -91 2034 -91 \n","Q 1263 -91 848 271 \n","Q 434 634 434 1313 \n","Q 434 1759 690 2068 \n","Q 947 2378 1403 2484 \n","z\n","M 1172 3481 \n","Q 1172 3119 1398 2916 \n","Q 1625 2713 2034 2713 \n","Q 2441 2713 2670 2916 \n","Q 2900 3119 2900 3481 \n","Q 2900 3844 2670 4047 \n","Q 2441 4250 2034 4250 \n","Q 1625 4250 1398 4047 \n","Q 1172 3844 1172 3481 \n","z\n","\" transform=\"scale(0.015625)\"/>\n","       </defs>\n","       <use xlink:href=\"#DejaVuSans-38\"/>\n","       <use xlink:href=\"#DejaVuSans-30\" x=\"63.623047\"/>\n","      </g>\n","     </g>\n","    </g>\n","    <g id=\"xtick_5\">\n","     <g id=\"line2d_9\">\n","      <path d=\"M 246.678125 143.1 \n","L 246.678125 7.2 \n","\" clip-path=\"url(#paf1508e7f6)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n","     </g>\n","     <g id=\"line2d_10\">\n","      <g>\n","       <use xlink:href=\"#m3a97404132\" x=\"246.678125\" y=\"143.1\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n","      </g>\n","     </g>\n","     <g id=\"text_5\">\n","      <!-- 100 -->\n","      <g transform=\"translate(237.134375 157.698438)scale(0.1 -0.1)\">\n","       <defs>\n","        <path id=\"DejaVuSans-31\" d=\"M 794 531 \n","L 1825 531 \n","L 1825 4091 \n","L 703 3866 \n","L 703 4441 \n","L 1819 4666 \n","L 2450 4666 \n","L 2450 531 \n","L 3481 531 \n","L 3481 0 \n","L 794 0 \n","L 794 531 \n","z\n","\" transform=\"scale(0.015625)\"/>\n","       </defs>\n","       <use xlink:href=\"#DejaVuSans-31\"/>\n","       <use xlink:href=\"#DejaVuSans-30\" x=\"63.623047\"/>\n","       <use xlink:href=\"#DejaVuSans-30\" x=\"127.246094\"/>\n","      </g>\n","     </g>\n","    </g>\n","    <g id=\"text_6\">\n","     <!-- epochs -->\n","     <g transform=\"translate(131.195312 171.376563)scale(0.1 -0.1)\">\n","      <defs>\n","       <path id=\"DejaVuSans-65\" d=\"M 3597 1894 \n","L 3597 1613 \n","L 953 1613 \n","Q 991 1019 1311 708 \n","Q 1631 397 2203 397 \n","Q 2534 397 2845 478 \n","Q 3156 559 3463 722 \n","L 3463 178 \n","Q 3153 47 2828 -22 \n","Q 2503 -91 2169 -91 \n","Q 1331 -91 842 396 \n","Q 353 884 353 1716 \n","Q 353 2575 817 3079 \n","Q 1281 3584 2069 3584 \n","Q 2775 3584 3186 3129 \n","Q 3597 2675 3597 1894 \n","z\n","M 3022 2063 \n","Q 3016 2534 2758 2815 \n","Q 2500 3097 2075 3097 \n","Q 1594 3097 1305 2825 \n","Q 1016 2553 972 2059 \n","L 3022 2063 \n","z\n","\" transform=\"scale(0.015625)\"/>\n","       <path id=\"DejaVuSans-70\" d=\"M 1159 525 \n","L 1159 -1331 \n","L 581 -1331 \n","L 581 3500 \n","L 1159 3500 \n","L 1159 2969 \n","Q 1341 3281 1617 3432 \n","Q 1894 3584 2278 3584 \n","Q 2916 3584 3314 3078 \n","Q 3713 2572 3713 1747 \n","Q 3713 922 3314 415 \n","Q 2916 -91 2278 -91 \n","Q 1894 -91 1617 61 \n","Q 1341 213 1159 525 \n","z\n","M 3116 1747 \n","Q 3116 2381 2855 2742 \n","Q 2594 3103 2138 3103 \n","Q 1681 3103 1420 2742 \n","Q 1159 2381 1159 1747 \n","Q 1159 1113 1420 752 \n","Q 1681 391 2138 391 \n","Q 2594 391 2855 752 \n","Q 3116 1113 3116 1747 \n","z\n","\" transform=\"scale(0.015625)\"/>\n","       <path id=\"DejaVuSans-6f\" d=\"M 1959 3097 \n","Q 1497 3097 1228 2736 \n","Q 959 2375 959 1747 \n","Q 959 1119 1226 758 \n","Q 1494 397 1959 397 \n","Q 2419 397 2687 759 \n","Q 2956 1122 2956 1747 \n","Q 2956 2369 2687 2733 \n","Q 2419 3097 1959 3097 \n","z\n","M 1959 3584 \n","Q 2709 3584 3137 3096 \n","Q 3566 2609 3566 1747 \n","Q 3566 888 3137 398 \n","Q 2709 -91 1959 -91 \n","Q 1206 -91 779 398 \n","Q 353 888 353 1747 \n","Q 353 2609 779 3096 \n","Q 1206 3584 1959 3584 \n","z\n","\" transform=\"scale(0.015625)\"/>\n","       <path id=\"DejaVuSans-63\" d=\"M 3122 3366 \n","L 3122 2828 \n","Q 2878 2963 2633 3030 \n","Q 2388 3097 2138 3097 \n","Q 1578 3097 1268 2742 \n","Q 959 2388 959 1747 \n","Q 959 1106 1268 751 \n","Q 1578 397 2138 397 \n","Q 2388 397 2633 464 \n","Q 2878 531 3122 666 \n","L 3122 134 \n","Q 2881 22 2623 -34 \n","Q 2366 -91 2075 -91 \n","Q 1284 -91 818 406 \n","Q 353 903 353 1747 \n","Q 353 2603 823 3093 \n","Q 1294 3584 2113 3584 \n","Q 2378 3584 2631 3529 \n","Q 2884 3475 3122 3366 \n","z\n","\" transform=\"scale(0.015625)\"/>\n","       <path id=\"DejaVuSans-68\" d=\"M 3513 2113 \n","L 3513 0 \n","L 2938 0 \n","L 2938 2094 \n","Q 2938 2591 2744 2837 \n","Q 2550 3084 2163 3084 \n","Q 1697 3084 1428 2787 \n","Q 1159 2491 1159 1978 \n","L 1159 0 \n","L 581 0 \n","L 581 4863 \n","L 1159 4863 \n","L 1159 2956 \n","Q 1366 3272 1645 3428 \n","Q 1925 3584 2291 3584 \n","Q 2894 3584 3203 3211 \n","Q 3513 2838 3513 2113 \n","z\n","\" transform=\"scale(0.015625)\"/>\n","       <path id=\"DejaVuSans-73\" d=\"M 2834 3397 \n","L 2834 2853 \n","Q 2591 2978 2328 3040 \n","Q 2066 3103 1784 3103 \n","Q 1356 3103 1142 2972 \n","Q 928 2841 928 2578 \n","Q 928 2378 1081 2264 \n","Q 1234 2150 1697 2047 \n","L 1894 2003 \n","Q 2506 1872 2764 1633 \n","Q 3022 1394 3022 966 \n","Q 3022 478 2636 193 \n","Q 2250 -91 1575 -91 \n","Q 1294 -91 989 -36 \n","Q 684 19 347 128 \n","L 347 722 \n","Q 666 556 975 473 \n","Q 1284 391 1588 391 \n","Q 1994 391 2212 530 \n","Q 2431 669 2431 922 \n","Q 2431 1156 2273 1281 \n","Q 2116 1406 1581 1522 \n","L 1381 1569 \n","Q 847 1681 609 1914 \n","Q 372 2147 372 2553 \n","Q 372 3047 722 3315 \n","Q 1072 3584 1716 3584 \n","Q 2034 3584 2315 3537 \n","Q 2597 3491 2834 3397 \n","z\n","\" transform=\"scale(0.015625)\"/>\n","      </defs>\n","      <use xlink:href=\"#DejaVuSans-65\"/>\n","      <use xlink:href=\"#DejaVuSans-70\" x=\"61.523438\"/>\n","      <use xlink:href=\"#DejaVuSans-6f\" x=\"125\"/>\n","      <use xlink:href=\"#DejaVuSans-63\" x=\"186.181641\"/>\n","      <use xlink:href=\"#DejaVuSans-68\" x=\"241.162109\"/>\n","      <use xlink:href=\"#DejaVuSans-73\" x=\"304.541016\"/>\n","     </g>\n","    </g>\n","   </g>\n","   <g id=\"matplotlib.axis_2\">\n","    <g id=\"ytick_1\">\n","     <g id=\"line2d_11\">\n","      <path d=\"M 51.378125 123.123527 \n","L 246.678125 123.123527 \n","\" clip-path=\"url(#paf1508e7f6)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n","     </g>\n","     <g id=\"line2d_12\">\n","      <defs>\n","       <path id=\"m35267bf620\" d=\"M 0 0 \n","L -3.5 0 \n","\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n","      </defs>\n","      <g>\n","       <use xlink:href=\"#m35267bf620\" x=\"51.378125\" y=\"123.123527\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n","      </g>\n","     </g>\n","     <g id=\"text_7\">\n","      <!-- $\\mathdefault{10^{-5}}$ -->\n","      <g transform=\"translate(20.878125 126.922746)scale(0.1 -0.1)\">\n","       <defs>\n","        <path id=\"DejaVuSans-2212\" d=\"M 678 2272 \n","L 4684 2272 \n","L 4684 1741 \n","L 678 1741 \n","L 678 2272 \n","z\n","\" transform=\"scale(0.015625)\"/>\n","        <path id=\"DejaVuSans-35\" d=\"M 691 4666 \n","L 3169 4666 \n","L 3169 4134 \n","L 1269 4134 \n","L 1269 2991 \n","Q 1406 3038 1543 3061 \n","Q 1681 3084 1819 3084 \n","Q 2600 3084 3056 2656 \n","Q 3513 2228 3513 1497 \n","Q 3513 744 3044 326 \n","Q 2575 -91 1722 -91 \n","Q 1428 -91 1123 -41 \n","Q 819 9 494 109 \n","L 494 744 \n","Q 775 591 1075 516 \n","Q 1375 441 1709 441 \n","Q 2250 441 2565 725 \n","Q 2881 1009 2881 1497 \n","Q 2881 1984 2565 2268 \n","Q 2250 2553 1709 2553 \n","Q 1456 2553 1204 2497 \n","Q 953 2441 691 2322 \n","L 691 4666 \n","z\n","\" transform=\"scale(0.015625)\"/>\n","       </defs>\n","       <use xlink:href=\"#DejaVuSans-31\" transform=\"translate(0 0.684375)\"/>\n","       <use xlink:href=\"#DejaVuSans-30\" transform=\"translate(63.623047 0.684375)\"/>\n","       <use xlink:href=\"#DejaVuSans-2212\" transform=\"translate(128.203125 38.965625)scale(0.7)\"/>\n","       <use xlink:href=\"#DejaVuSans-35\" transform=\"translate(186.855469 38.965625)scale(0.7)\"/>\n","      </g>\n","     </g>\n","    </g>\n","    <g id=\"ytick_2\">\n","     <g id=\"line2d_13\">\n","      <path d=\"M 51.378125 91.281762 \n","L 246.678125 91.281762 \n","\" clip-path=\"url(#paf1508e7f6)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n","     </g>\n","     <g id=\"line2d_14\">\n","      <g>\n","       <use xlink:href=\"#m35267bf620\" x=\"51.378125\" y=\"91.281762\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n","      </g>\n","     </g>\n","     <g id=\"text_8\">\n","      <!-- $\\mathdefault{10^{-3}}$ -->\n","      <g transform=\"translate(20.878125 95.080981)scale(0.1 -0.1)\">\n","       <defs>\n","        <path id=\"DejaVuSans-33\" d=\"M 2597 2516 \n","Q 3050 2419 3304 2112 \n","Q 3559 1806 3559 1356 \n","Q 3559 666 3084 287 \n","Q 2609 -91 1734 -91 \n","Q 1441 -91 1130 -33 \n","Q 819 25 488 141 \n","L 488 750 \n","Q 750 597 1062 519 \n","Q 1375 441 1716 441 \n","Q 2309 441 2620 675 \n","Q 2931 909 2931 1356 \n","Q 2931 1769 2642 2001 \n","Q 2353 2234 1838 2234 \n","L 1294 2234 \n","L 1294 2753 \n","L 1863 2753 \n","Q 2328 2753 2575 2939 \n","Q 2822 3125 2822 3475 \n","Q 2822 3834 2567 4026 \n","Q 2313 4219 1838 4219 \n","Q 1578 4219 1281 4162 \n","Q 984 4106 628 3988 \n","L 628 4550 \n","Q 988 4650 1302 4700 \n","Q 1616 4750 1894 4750 \n","Q 2613 4750 3031 4423 \n","Q 3450 4097 3450 3541 \n","Q 3450 3153 3228 2886 \n","Q 3006 2619 2597 2516 \n","z\n","\" transform=\"scale(0.015625)\"/>\n","       </defs>\n","       <use xlink:href=\"#DejaVuSans-31\" transform=\"translate(0 0.765625)\"/>\n","       <use xlink:href=\"#DejaVuSans-30\" transform=\"translate(63.623047 0.765625)\"/>\n","       <use xlink:href=\"#DejaVuSans-2212\" transform=\"translate(128.203125 39.046875)scale(0.7)\"/>\n","       <use xlink:href=\"#DejaVuSans-33\" transform=\"translate(186.855469 39.046875)scale(0.7)\"/>\n","      </g>\n","     </g>\n","    </g>\n","    <g id=\"ytick_3\">\n","     <g id=\"line2d_15\">\n","      <path d=\"M 51.378125 59.439996 \n","L 246.678125 59.439996 \n","\" clip-path=\"url(#paf1508e7f6)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n","     </g>\n","     <g id=\"line2d_16\">\n","      <g>\n","       <use xlink:href=\"#m35267bf620\" x=\"51.378125\" y=\"59.439996\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n","      </g>\n","     </g>\n","     <g id=\"text_9\">\n","      <!-- $\\mathdefault{10^{-1}}$ -->\n","      <g transform=\"translate(20.878125 63.239215)scale(0.1 -0.1)\">\n","       <use xlink:href=\"#DejaVuSans-31\" transform=\"translate(0 0.684375)\"/>\n","       <use xlink:href=\"#DejaVuSans-30\" transform=\"translate(63.623047 0.684375)\"/>\n","       <use xlink:href=\"#DejaVuSans-2212\" transform=\"translate(128.203125 38.965625)scale(0.7)\"/>\n","       <use xlink:href=\"#DejaVuSans-31\" transform=\"translate(186.855469 38.965625)scale(0.7)\"/>\n","      </g>\n","     </g>\n","    </g>\n","    <g id=\"ytick_4\">\n","     <g id=\"line2d_17\">\n","      <path d=\"M 51.378125 27.598231 \n","L 246.678125 27.598231 \n","\" clip-path=\"url(#paf1508e7f6)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n","     </g>\n","     <g id=\"line2d_18\">\n","      <g>\n","       <use xlink:href=\"#m35267bf620\" x=\"51.378125\" y=\"27.598231\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n","      </g>\n","     </g>\n","     <g id=\"text_10\">\n","      <!-- $\\mathdefault{10^{1}}$ -->\n","      <g transform=\"translate(26.778125 31.397449)scale(0.1 -0.1)\">\n","       <use xlink:href=\"#DejaVuSans-31\" transform=\"translate(0 0.684375)\"/>\n","       <use xlink:href=\"#DejaVuSans-30\" transform=\"translate(63.623047 0.684375)\"/>\n","       <use xlink:href=\"#DejaVuSans-31\" transform=\"translate(128.203125 38.965625)scale(0.7)\"/>\n","      </g>\n","     </g>\n","    </g>\n","    <g id=\"text_11\">\n","     <!-- loss -->\n","     <g transform=\"translate(14.798437 84.807812)rotate(-90)scale(0.1 -0.1)\">\n","      <defs>\n","       <path id=\"DejaVuSans-6c\" d=\"M 603 4863 \n","L 1178 4863 \n","L 1178 0 \n","L 603 0 \n","L 603 4863 \n","z\n","\" transform=\"scale(0.015625)\"/>\n","      </defs>\n","      <use xlink:href=\"#DejaVuSans-6c\"/>\n","      <use xlink:href=\"#DejaVuSans-6f\" x=\"27.783203\"/>\n","      <use xlink:href=\"#DejaVuSans-73\" x=\"88.964844\"/>\n","      <use xlink:href=\"#DejaVuSans-73\" x=\"141.064453\"/>\n","     </g>\n","    </g>\n","   </g>\n","   <g id=\"line2d_19\">\n","    <path d=\"M 51.378125 26.106104 \n","L 61.657072 35.21742 \n","L 71.93602 43.468449 \n","L 82.214967 51.117298 \n","L 92.493914 58.352648 \n","L 102.772862 65.218746 \n","L 113.051809 71.755366 \n","L 123.330757 77.981748 \n","L 133.609704 83.887567 \n","L 143.888651 89.524081 \n","L 154.167599 94.890499 \n","L 164.446546 100.052997 \n","L 174.725493 105.018093 \n","L 185.004441 109.832648 \n","L 195.283388 114.5259 \n","L 205.562336 119.116875 \n","L 215.841283 123.634023 \n","L 226.12023 128.098305 \n","L 236.399178 132.53015 \n","L 246.678125 136.922727 \n","\" clip-path=\"url(#paf1508e7f6)\" style=\"fill: none; stroke: #1f77b4; stroke-width: 1.5; stroke-linecap: square\"/>\n","   </g>\n","   <g id=\"line2d_20\">\n","    <path d=\"M 51.378125 13.38739 \n","L 61.657072 13.41182 \n","L 71.93602 13.407508 \n","L 82.214967 13.399341 \n","L 92.493914 13.392222 \n","L 102.772862 13.386754 \n","L 113.051809 13.383239 \n","L 123.330757 13.380876 \n","L 133.609704 13.379493 \n","L 143.888651 13.378573 \n","L 154.167599 13.378049 \n","L 164.446546 13.377708 \n","L 174.725493 13.377516 \n","L 185.004441 13.377404 \n","L 195.283388 13.377345 \n","L 205.562336 13.377309 \n","L 215.841283 13.377291 \n","L 226.12023 13.377279 \n","L 236.399178 13.377276 \n","L 246.678125 13.377273 \n","\" clip-path=\"url(#paf1508e7f6)\" style=\"fill: none; stroke-dasharray: 5.55,2.4; stroke-dashoffset: 0; stroke: #bf00bf; stroke-width: 1.5\"/>\n","   </g>\n","   <g id=\"patch_3\">\n","    <path d=\"M 51.378125 143.1 \n","L 51.378125 7.2 \n","\" style=\"fill: none; stroke: #000000; stroke-width: 0.8; stroke-linejoin: miter; stroke-linecap: square\"/>\n","   </g>\n","   <g id=\"patch_4\">\n","    <path d=\"M 246.678125 143.1 \n","L 246.678125 7.2 \n","\" style=\"fill: none; stroke: #000000; stroke-width: 0.8; stroke-linejoin: miter; stroke-linecap: square\"/>\n","   </g>\n","   <g id=\"patch_5\">\n","    <path d=\"M 51.378125 143.1 \n","L 246.678125 143.1 \n","\" style=\"fill: none; stroke: #000000; stroke-width: 0.8; stroke-linejoin: miter; stroke-linecap: square\"/>\n","   </g>\n","   <g id=\"patch_6\">\n","    <path d=\"M 51.378125 7.2 \n","L 246.678125 7.2 \n","\" style=\"fill: none; stroke: #000000; stroke-width: 0.8; stroke-linejoin: miter; stroke-linecap: square\"/>\n","   </g>\n","   <g id=\"legend_1\">\n","    <g id=\"patch_7\">\n","     <path d=\"M 58.378125 138.1 \n","L 113.653125 138.1 \n","Q 115.653125 138.1 115.653125 136.1 \n","L 115.653125 107.74375 \n","Q 115.653125 105.74375 113.653125 105.74375 \n","L 58.378125 105.74375 \n","Q 56.378125 105.74375 56.378125 107.74375 \n","L 56.378125 136.1 \n","Q 56.378125 138.1 58.378125 138.1 \n","z\n","\" style=\"fill: #ffffff; opacity: 0.8; stroke: #cccccc; stroke-linejoin: miter\"/>\n","    </g>\n","    <g id=\"line2d_21\">\n","     <path d=\"M 60.378125 113.842187 \n","L 70.378125 113.842187 \n","L 80.378125 113.842187 \n","\" style=\"fill: none; stroke: #1f77b4; stroke-width: 1.5; stroke-linecap: square\"/>\n","    </g>\n","    <g id=\"text_12\">\n","     <!-- train -->\n","     <g transform=\"translate(88.378125 117.342187)scale(0.1 -0.1)\">\n","      <defs>\n","       <path id=\"DejaVuSans-74\" d=\"M 1172 4494 \n","L 1172 3500 \n","L 2356 3500 \n","L 2356 3053 \n","L 1172 3053 \n","L 1172 1153 \n","Q 1172 725 1289 603 \n","Q 1406 481 1766 481 \n","L 2356 481 \n","L 2356 0 \n","L 1766 0 \n","Q 1100 0 847 248 \n","Q 594 497 594 1153 \n","L 594 3053 \n","L 172 3053 \n","L 172 3500 \n","L 594 3500 \n","L 594 4494 \n","L 1172 4494 \n","z\n","\" transform=\"scale(0.015625)\"/>\n","       <path id=\"DejaVuSans-72\" d=\"M 2631 2963 \n","Q 2534 3019 2420 3045 \n","Q 2306 3072 2169 3072 \n","Q 1681 3072 1420 2755 \n","Q 1159 2438 1159 1844 \n","L 1159 0 \n","L 581 0 \n","L 581 3500 \n","L 1159 3500 \n","L 1159 2956 \n","Q 1341 3275 1631 3429 \n","Q 1922 3584 2338 3584 \n","Q 2397 3584 2469 3576 \n","Q 2541 3569 2628 3553 \n","L 2631 2963 \n","z\n","\" transform=\"scale(0.015625)\"/>\n","       <path id=\"DejaVuSans-61\" d=\"M 2194 1759 \n","Q 1497 1759 1228 1600 \n","Q 959 1441 959 1056 \n","Q 959 750 1161 570 \n","Q 1363 391 1709 391 \n","Q 2188 391 2477 730 \n","Q 2766 1069 2766 1631 \n","L 2766 1759 \n","L 2194 1759 \n","z\n","M 3341 1997 \n","L 3341 0 \n","L 2766 0 \n","L 2766 531 \n","Q 2569 213 2275 61 \n","Q 1981 -91 1556 -91 \n","Q 1019 -91 701 211 \n","Q 384 513 384 1019 \n","Q 384 1609 779 1909 \n","Q 1175 2209 1959 2209 \n","L 2766 2209 \n","L 2766 2266 \n","Q 2766 2663 2505 2880 \n","Q 2244 3097 1772 3097 \n","Q 1472 3097 1187 3025 \n","Q 903 2953 641 2809 \n","L 641 3341 \n","Q 956 3463 1253 3523 \n","Q 1550 3584 1831 3584 \n","Q 2591 3584 2966 3190 \n","Q 3341 2797 3341 1997 \n","z\n","\" transform=\"scale(0.015625)\"/>\n","       <path id=\"DejaVuSans-69\" d=\"M 603 3500 \n","L 1178 3500 \n","L 1178 0 \n","L 603 0 \n","L 603 3500 \n","z\n","M 603 4863 \n","L 1178 4863 \n","L 1178 4134 \n","L 603 4134 \n","L 603 4863 \n","z\n","\" transform=\"scale(0.015625)\"/>\n","       <path id=\"DejaVuSans-6e\" d=\"M 3513 2113 \n","L 3513 0 \n","L 2938 0 \n","L 2938 2094 \n","Q 2938 2591 2744 2837 \n","Q 2550 3084 2163 3084 \n","Q 1697 3084 1428 2787 \n","Q 1159 2491 1159 1978 \n","L 1159 0 \n","L 581 0 \n","L 581 3500 \n","L 1159 3500 \n","L 1159 2956 \n","Q 1366 3272 1645 3428 \n","Q 1925 3584 2291 3584 \n","Q 2894 3584 3203 3211 \n","Q 3513 2838 3513 2113 \n","z\n","\" transform=\"scale(0.015625)\"/>\n","      </defs>\n","      <use xlink:href=\"#DejaVuSans-74\"/>\n","      <use xlink:href=\"#DejaVuSans-72\" x=\"39.208984\"/>\n","      <use xlink:href=\"#DejaVuSans-61\" x=\"80.322266\"/>\n","      <use xlink:href=\"#DejaVuSans-69\" x=\"141.601562\"/>\n","      <use xlink:href=\"#DejaVuSans-6e\" x=\"169.384766\"/>\n","     </g>\n","    </g>\n","    <g id=\"line2d_22\">\n","     <path d=\"M 60.378125 128.520312 \n","L 70.378125 128.520312 \n","L 80.378125 128.520312 \n","\" style=\"fill: none; stroke-dasharray: 5.55,2.4; stroke-dashoffset: 0; stroke: #bf00bf; stroke-width: 1.5\"/>\n","    </g>\n","    <g id=\"text_13\">\n","     <!-- test -->\n","     <g transform=\"translate(88.378125 132.020312)scale(0.1 -0.1)\">\n","      <use xlink:href=\"#DejaVuSans-74\"/>\n","      <use xlink:href=\"#DejaVuSans-65\" x=\"39.208984\"/>\n","      <use xlink:href=\"#DejaVuSans-73\" x=\"100.732422\"/>\n","      <use xlink:href=\"#DejaVuSans-74\" x=\"152.832031\"/>\n","     </g>\n","    </g>\n","   </g>\n","  </g>\n"," </g>\n"," <defs>\n","  <clipPath id=\"paf1508e7f6\">\n","   <rect x=\"51.378125\" y=\"7.2\" width=\"195.3\" height=\"135.9\"/>\n","  </clipPath>\n"," </defs>\n","</svg>\n"],"text/plain":["<Figure size 252x180 with 1 Axes>"]},"metadata":{"needs_background":"light"},"output_type":"display_data"}],"source":["train(lambd=0)"],"id":"jYYnQcyphB9D"},{"cell_type":"markdown","metadata":{"origin_pos":20,"id":"FkvOxC-MhB9D"},"source":["### [**使用权重衰减**]\n","\n","下面，我们使用权重衰减来运行代码。\n","注意，在这里训练误差增大，但测试误差减小。\n","这正是我们期望从正则化中得到的效果。\n"],"id":"FkvOxC-MhB9D"},{"cell_type":"code","execution_count":null,"metadata":{"execution":{"iopub.execute_input":"2022-07-31T02:22:55.835902Z","iopub.status.busy":"2022-07-31T02:22:55.835673Z","iopub.status.idle":"2022-07-31T02:23:01.979648Z","shell.execute_reply":"2022-07-31T02:23:01.978765Z"},"origin_pos":21,"tab":["pytorch"],"outputId":"a9e1657b-9711-4d71-e957-c4ae96b7a725","id":"SxRzn_szhB9D"},"outputs":[{"name":"stdout","output_type":"stream","text":["w的L2范数是： 0.3558707535266876\n"]},{"data":{"image/svg+xml":["<?xml version=\"1.0\" encoding=\"utf-8\" standalone=\"no\"?>\n","<!DOCTYPE svg PUBLIC \"-//W3C//DTD SVG 1.1//EN\"\n","  \"http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd\">\n","<svg xmlns:xlink=\"http://www.w3.org/1999/xlink\" width=\"263.421875pt\" height=\"182.600266pt\" viewBox=\"0 0 263.421875 182.600266\" xmlns=\"http://www.w3.org/2000/svg\" version=\"1.1\">\n"," <metadata>\n","  <rdf:RDF xmlns:dc=\"http://purl.org/dc/elements/1.1/\" xmlns:cc=\"http://creativecommons.org/ns#\" xmlns:rdf=\"http://www.w3.org/1999/02/22-rdf-syntax-ns#\">\n","   <cc:Work>\n","    <dc:type rdf:resource=\"http://purl.org/dc/dcmitype/StillImage\"/>\n","    <dc:date>2022-07-31T02:23:01.916702</dc:date>\n","    <dc:format>image/svg+xml</dc:format>\n","    <dc:creator>\n","     <cc:Agent>\n","      <dc:title>Matplotlib v3.5.1, https://matplotlib.org/</dc:title>\n","     </cc:Agent>\n","    </dc:creator>\n","   </cc:Work>\n","  </rdf:RDF>\n"," </metadata>\n"," <defs>\n","  <style type=\"text/css\">*{stroke-linejoin: round; stroke-linecap: butt}</style>\n"," </defs>\n"," <g id=\"figure_1\">\n","  <g id=\"patch_1\">\n","   <path d=\"M 0 182.600266 \n","L 263.421875 182.600266 \n","L 263.421875 -0 \n","L 0 -0 \n","L 0 182.600266 \n","z\n","\" style=\"fill: none\"/>\n","  </g>\n","  <g id=\"axes_1\">\n","   <g id=\"patch_2\">\n","    <path d=\"M 51.378125 145.044016 \n","L 246.678125 145.044016 \n","L 246.678125 9.144016 \n","L 51.378125 9.144016 \n","z\n","\" style=\"fill: #ffffff\"/>\n","   </g>\n","   <g id=\"matplotlib.axis_1\">\n","    <g id=\"xtick_1\">\n","     <g id=\"line2d_1\">\n","      <path d=\"M 82.214967 145.044016 \n","L 82.214967 9.144016 \n","\" clip-path=\"url(#pd83c5e35c7)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n","     </g>\n","     <g id=\"line2d_2\">\n","      <defs>\n","       <path id=\"mc4b0c12c4b\" d=\"M 0 0 \n","L 0 3.5 \n","\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n","      </defs>\n","      <g>\n","       <use xlink:href=\"#mc4b0c12c4b\" x=\"82.214967\" y=\"145.044016\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n","      </g>\n","     </g>\n","     <g id=\"text_1\">\n","      <!-- 20 -->\n","      <g transform=\"translate(75.852467 159.642454)scale(0.1 -0.1)\">\n","       <defs>\n","        <path id=\"DejaVuSans-32\" d=\"M 1228 531 \n","L 3431 531 \n","L 3431 0 \n","L 469 0 \n","L 469 531 \n","Q 828 903 1448 1529 \n","Q 2069 2156 2228 2338 \n","Q 2531 2678 2651 2914 \n","Q 2772 3150 2772 3378 \n","Q 2772 3750 2511 3984 \n","Q 2250 4219 1831 4219 \n","Q 1534 4219 1204 4116 \n","Q 875 4013 500 3803 \n","L 500 4441 \n","Q 881 4594 1212 4672 \n","Q 1544 4750 1819 4750 \n","Q 2544 4750 2975 4387 \n","Q 3406 4025 3406 3419 \n","Q 3406 3131 3298 2873 \n","Q 3191 2616 2906 2266 \n","Q 2828 2175 2409 1742 \n","Q 1991 1309 1228 531 \n","z\n","\" transform=\"scale(0.015625)\"/>\n","        <path id=\"DejaVuSans-30\" d=\"M 2034 4250 \n","Q 1547 4250 1301 3770 \n","Q 1056 3291 1056 2328 \n","Q 1056 1369 1301 889 \n","Q 1547 409 2034 409 \n","Q 2525 409 2770 889 \n","Q 3016 1369 3016 2328 \n","Q 3016 3291 2770 3770 \n","Q 2525 4250 2034 4250 \n","z\n","M 2034 4750 \n","Q 2819 4750 3233 4129 \n","Q 3647 3509 3647 2328 \n","Q 3647 1150 3233 529 \n","Q 2819 -91 2034 -91 \n","Q 1250 -91 836 529 \n","Q 422 1150 422 2328 \n","Q 422 3509 836 4129 \n","Q 1250 4750 2034 4750 \n","z\n","\" transform=\"scale(0.015625)\"/>\n","       </defs>\n","       <use xlink:href=\"#DejaVuSans-32\"/>\n","       <use xlink:href=\"#DejaVuSans-30\" x=\"63.623047\"/>\n","      </g>\n","     </g>\n","    </g>\n","    <g id=\"xtick_2\">\n","     <g id=\"line2d_3\">\n","      <path d=\"M 123.330757 145.044016 \n","L 123.330757 9.144016 \n","\" clip-path=\"url(#pd83c5e35c7)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n","     </g>\n","     <g id=\"line2d_4\">\n","      <g>\n","       <use xlink:href=\"#mc4b0c12c4b\" x=\"123.330757\" y=\"145.044016\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n","      </g>\n","     </g>\n","     <g id=\"text_2\">\n","      <!-- 40 -->\n","      <g transform=\"translate(116.968257 159.642454)scale(0.1 -0.1)\">\n","       <defs>\n","        <path id=\"DejaVuSans-34\" d=\"M 2419 4116 \n","L 825 1625 \n","L 2419 1625 \n","L 2419 4116 \n","z\n","M 2253 4666 \n","L 3047 4666 \n","L 3047 1625 \n","L 3713 1625 \n","L 3713 1100 \n","L 3047 1100 \n","L 3047 0 \n","L 2419 0 \n","L 2419 1100 \n","L 313 1100 \n","L 313 1709 \n","L 2253 4666 \n","z\n","\" transform=\"scale(0.015625)\"/>\n","       </defs>\n","       <use xlink:href=\"#DejaVuSans-34\"/>\n","       <use xlink:href=\"#DejaVuSans-30\" x=\"63.623047\"/>\n","      </g>\n","     </g>\n","    </g>\n","    <g id=\"xtick_3\">\n","     <g id=\"line2d_5\">\n","      <path d=\"M 164.446546 145.044016 \n","L 164.446546 9.144016 \n","\" clip-path=\"url(#pd83c5e35c7)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n","     </g>\n","     <g id=\"line2d_6\">\n","      <g>\n","       <use xlink:href=\"#mc4b0c12c4b\" x=\"164.446546\" y=\"145.044016\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n","      </g>\n","     </g>\n","     <g id=\"text_3\">\n","      <!-- 60 -->\n","      <g transform=\"translate(158.084046 159.642454)scale(0.1 -0.1)\">\n","       <defs>\n","        <path id=\"DejaVuSans-36\" d=\"M 2113 2584 \n","Q 1688 2584 1439 2293 \n","Q 1191 2003 1191 1497 \n","Q 1191 994 1439 701 \n","Q 1688 409 2113 409 \n","Q 2538 409 2786 701 \n","Q 3034 994 3034 1497 \n","Q 3034 2003 2786 2293 \n","Q 2538 2584 2113 2584 \n","z\n","M 3366 4563 \n","L 3366 3988 \n","Q 3128 4100 2886 4159 \n","Q 2644 4219 2406 4219 \n","Q 1781 4219 1451 3797 \n","Q 1122 3375 1075 2522 \n","Q 1259 2794 1537 2939 \n","Q 1816 3084 2150 3084 \n","Q 2853 3084 3261 2657 \n","Q 3669 2231 3669 1497 \n","Q 3669 778 3244 343 \n","Q 2819 -91 2113 -91 \n","Q 1303 -91 875 529 \n","Q 447 1150 447 2328 \n","Q 447 3434 972 4092 \n","Q 1497 4750 2381 4750 \n","Q 2619 4750 2861 4703 \n","Q 3103 4656 3366 4563 \n","z\n","\" transform=\"scale(0.015625)\"/>\n","       </defs>\n","       <use xlink:href=\"#DejaVuSans-36\"/>\n","       <use xlink:href=\"#DejaVuSans-30\" x=\"63.623047\"/>\n","      </g>\n","     </g>\n","    </g>\n","    <g id=\"xtick_4\">\n","     <g id=\"line2d_7\">\n","      <path d=\"M 205.562336 145.044016 \n","L 205.562336 9.144016 \n","\" clip-path=\"url(#pd83c5e35c7)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n","     </g>\n","     <g id=\"line2d_8\">\n","      <g>\n","       <use xlink:href=\"#mc4b0c12c4b\" x=\"205.562336\" y=\"145.044016\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n","      </g>\n","     </g>\n","     <g id=\"text_4\">\n","      <!-- 80 -->\n","      <g transform=\"translate(199.199836 159.642454)scale(0.1 -0.1)\">\n","       <defs>\n","        <path id=\"DejaVuSans-38\" d=\"M 2034 2216 \n","Q 1584 2216 1326 1975 \n","Q 1069 1734 1069 1313 \n","Q 1069 891 1326 650 \n","Q 1584 409 2034 409 \n","Q 2484 409 2743 651 \n","Q 3003 894 3003 1313 \n","Q 3003 1734 2745 1975 \n","Q 2488 2216 2034 2216 \n","z\n","M 1403 2484 \n","Q 997 2584 770 2862 \n","Q 544 3141 544 3541 \n","Q 544 4100 942 4425 \n","Q 1341 4750 2034 4750 \n","Q 2731 4750 3128 4425 \n","Q 3525 4100 3525 3541 \n","Q 3525 3141 3298 2862 \n","Q 3072 2584 2669 2484 \n","Q 3125 2378 3379 2068 \n","Q 3634 1759 3634 1313 \n","Q 3634 634 3220 271 \n","Q 2806 -91 2034 -91 \n","Q 1263 -91 848 271 \n","Q 434 634 434 1313 \n","Q 434 1759 690 2068 \n","Q 947 2378 1403 2484 \n","z\n","M 1172 3481 \n","Q 1172 3119 1398 2916 \n","Q 1625 2713 2034 2713 \n","Q 2441 2713 2670 2916 \n","Q 2900 3119 2900 3481 \n","Q 2900 3844 2670 4047 \n","Q 2441 4250 2034 4250 \n","Q 1625 4250 1398 4047 \n","Q 1172 3844 1172 3481 \n","z\n","\" transform=\"scale(0.015625)\"/>\n","       </defs>\n","       <use xlink:href=\"#DejaVuSans-38\"/>\n","       <use xlink:href=\"#DejaVuSans-30\" x=\"63.623047\"/>\n","      </g>\n","     </g>\n","    </g>\n","    <g id=\"xtick_5\">\n","     <g id=\"line2d_9\">\n","      <path d=\"M 246.678125 145.044016 \n","L 246.678125 9.144016 \n","\" clip-path=\"url(#pd83c5e35c7)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n","     </g>\n","     <g id=\"line2d_10\">\n","      <g>\n","       <use xlink:href=\"#mc4b0c12c4b\" x=\"246.678125\" y=\"145.044016\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n","      </g>\n","     </g>\n","     <g id=\"text_5\">\n","      <!-- 100 -->\n","      <g transform=\"translate(237.134375 159.642454)scale(0.1 -0.1)\">\n","       <defs>\n","        <path id=\"DejaVuSans-31\" d=\"M 794 531 \n","L 1825 531 \n","L 1825 4091 \n","L 703 3866 \n","L 703 4441 \n","L 1819 4666 \n","L 2450 4666 \n","L 2450 531 \n","L 3481 531 \n","L 3481 0 \n","L 794 0 \n","L 794 531 \n","z\n","\" transform=\"scale(0.015625)\"/>\n","       </defs>\n","       <use xlink:href=\"#DejaVuSans-31\"/>\n","       <use xlink:href=\"#DejaVuSans-30\" x=\"63.623047\"/>\n","       <use xlink:href=\"#DejaVuSans-30\" x=\"127.246094\"/>\n","      </g>\n","     </g>\n","    </g>\n","    <g id=\"text_6\">\n","     <!-- epochs -->\n","     <g transform=\"translate(131.195312 173.320579)scale(0.1 -0.1)\">\n","      <defs>\n","       <path id=\"DejaVuSans-65\" d=\"M 3597 1894 \n","L 3597 1613 \n","L 953 1613 \n","Q 991 1019 1311 708 \n","Q 1631 397 2203 397 \n","Q 2534 397 2845 478 \n","Q 3156 559 3463 722 \n","L 3463 178 \n","Q 3153 47 2828 -22 \n","Q 2503 -91 2169 -91 \n","Q 1331 -91 842 396 \n","Q 353 884 353 1716 \n","Q 353 2575 817 3079 \n","Q 1281 3584 2069 3584 \n","Q 2775 3584 3186 3129 \n","Q 3597 2675 3597 1894 \n","z\n","M 3022 2063 \n","Q 3016 2534 2758 2815 \n","Q 2500 3097 2075 3097 \n","Q 1594 3097 1305 2825 \n","Q 1016 2553 972 2059 \n","L 3022 2063 \n","z\n","\" transform=\"scale(0.015625)\"/>\n","       <path id=\"DejaVuSans-70\" d=\"M 1159 525 \n","L 1159 -1331 \n","L 581 -1331 \n","L 581 3500 \n","L 1159 3500 \n","L 1159 2969 \n","Q 1341 3281 1617 3432 \n","Q 1894 3584 2278 3584 \n","Q 2916 3584 3314 3078 \n","Q 3713 2572 3713 1747 \n","Q 3713 922 3314 415 \n","Q 2916 -91 2278 -91 \n","Q 1894 -91 1617 61 \n","Q 1341 213 1159 525 \n","z\n","M 3116 1747 \n","Q 3116 2381 2855 2742 \n","Q 2594 3103 2138 3103 \n","Q 1681 3103 1420 2742 \n","Q 1159 2381 1159 1747 \n","Q 1159 1113 1420 752 \n","Q 1681 391 2138 391 \n","Q 2594 391 2855 752 \n","Q 3116 1113 3116 1747 \n","z\n","\" transform=\"scale(0.015625)\"/>\n","       <path id=\"DejaVuSans-6f\" d=\"M 1959 3097 \n","Q 1497 3097 1228 2736 \n","Q 959 2375 959 1747 \n","Q 959 1119 1226 758 \n","Q 1494 397 1959 397 \n","Q 2419 397 2687 759 \n","Q 2956 1122 2956 1747 \n","Q 2956 2369 2687 2733 \n","Q 2419 3097 1959 3097 \n","z\n","M 1959 3584 \n","Q 2709 3584 3137 3096 \n","Q 3566 2609 3566 1747 \n","Q 3566 888 3137 398 \n","Q 2709 -91 1959 -91 \n","Q 1206 -91 779 398 \n","Q 353 888 353 1747 \n","Q 353 2609 779 3096 \n","Q 1206 3584 1959 3584 \n","z\n","\" transform=\"scale(0.015625)\"/>\n","       <path id=\"DejaVuSans-63\" d=\"M 3122 3366 \n","L 3122 2828 \n","Q 2878 2963 2633 3030 \n","Q 2388 3097 2138 3097 \n","Q 1578 3097 1268 2742 \n","Q 959 2388 959 1747 \n","Q 959 1106 1268 751 \n","Q 1578 397 2138 397 \n","Q 2388 397 2633 464 \n","Q 2878 531 3122 666 \n","L 3122 134 \n","Q 2881 22 2623 -34 \n","Q 2366 -91 2075 -91 \n","Q 1284 -91 818 406 \n","Q 353 903 353 1747 \n","Q 353 2603 823 3093 \n","Q 1294 3584 2113 3584 \n","Q 2378 3584 2631 3529 \n","Q 2884 3475 3122 3366 \n","z\n","\" transform=\"scale(0.015625)\"/>\n","       <path id=\"DejaVuSans-68\" d=\"M 3513 2113 \n","L 3513 0 \n","L 2938 0 \n","L 2938 2094 \n","Q 2938 2591 2744 2837 \n","Q 2550 3084 2163 3084 \n","Q 1697 3084 1428 2787 \n","Q 1159 2491 1159 1978 \n","L 1159 0 \n","L 581 0 \n","L 581 4863 \n","L 1159 4863 \n","L 1159 2956 \n","Q 1366 3272 1645 3428 \n","Q 1925 3584 2291 3584 \n","Q 2894 3584 3203 3211 \n","Q 3513 2838 3513 2113 \n","z\n","\" transform=\"scale(0.015625)\"/>\n","       <path id=\"DejaVuSans-73\" d=\"M 2834 3397 \n","L 2834 2853 \n","Q 2591 2978 2328 3040 \n","Q 2066 3103 1784 3103 \n","Q 1356 3103 1142 2972 \n","Q 928 2841 928 2578 \n","Q 928 2378 1081 2264 \n","Q 1234 2150 1697 2047 \n","L 1894 2003 \n","Q 2506 1872 2764 1633 \n","Q 3022 1394 3022 966 \n","Q 3022 478 2636 193 \n","Q 2250 -91 1575 -91 \n","Q 1294 -91 989 -36 \n","Q 684 19 347 128 \n","L 347 722 \n","Q 666 556 975 473 \n","Q 1284 391 1588 391 \n","Q 1994 391 2212 530 \n","Q 2431 669 2431 922 \n","Q 2431 1156 2273 1281 \n","Q 2116 1406 1581 1522 \n","L 1381 1569 \n","Q 847 1681 609 1914 \n","Q 372 2147 372 2553 \n","Q 372 3047 722 3315 \n","Q 1072 3584 1716 3584 \n","Q 2034 3584 2315 3537 \n","Q 2597 3491 2834 3397 \n","z\n","\" transform=\"scale(0.015625)\"/>\n","      </defs>\n","      <use xlink:href=\"#DejaVuSans-65\"/>\n","      <use xlink:href=\"#DejaVuSans-70\" x=\"61.523438\"/>\n","      <use xlink:href=\"#DejaVuSans-6f\" x=\"125\"/>\n","      <use xlink:href=\"#DejaVuSans-63\" x=\"186.181641\"/>\n","      <use xlink:href=\"#DejaVuSans-68\" x=\"241.162109\"/>\n","      <use xlink:href=\"#DejaVuSans-73\" x=\"304.541016\"/>\n","     </g>\n","    </g>\n","   </g>\n","   <g id=\"matplotlib.axis_2\">\n","    <g id=\"ytick_1\">\n","     <g id=\"line2d_11\">\n","      <path d=\"M 51.378125 112.581014 \n","L 246.678125 112.581014 \n","\" clip-path=\"url(#pd83c5e35c7)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n","     </g>\n","     <g id=\"line2d_12\">\n","      <defs>\n","       <path id=\"ma8aaa4f60f\" d=\"M 0 0 \n","L -3.5 0 \n","\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n","      </defs>\n","      <g>\n","       <use xlink:href=\"#ma8aaa4f60f\" x=\"51.378125\" y=\"112.581014\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n","      </g>\n","     </g>\n","     <g id=\"text_7\">\n","      <!-- $\\mathdefault{10^{-2}}$ -->\n","      <g transform=\"translate(20.878125 116.380233)scale(0.1 -0.1)\">\n","       <defs>\n","        <path id=\"DejaVuSans-2212\" d=\"M 678 2272 \n","L 4684 2272 \n","L 4684 1741 \n","L 678 1741 \n","L 678 2272 \n","z\n","\" transform=\"scale(0.015625)\"/>\n","       </defs>\n","       <use xlink:href=\"#DejaVuSans-31\" transform=\"translate(0 0.765625)\"/>\n","       <use xlink:href=\"#DejaVuSans-30\" transform=\"translate(63.623047 0.765625)\"/>\n","       <use xlink:href=\"#DejaVuSans-2212\" transform=\"translate(128.203125 39.046875)scale(0.7)\"/>\n","       <use xlink:href=\"#DejaVuSans-32\" transform=\"translate(186.855469 39.046875)scale(0.7)\"/>\n","      </g>\n","     </g>\n","    </g>\n","    <g id=\"ytick_2\">\n","     <g id=\"line2d_13\">\n","      <path d=\"M 51.378125 61.790117 \n","L 246.678125 61.790117 \n","\" clip-path=\"url(#pd83c5e35c7)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n","     </g>\n","     <g id=\"line2d_14\">\n","      <g>\n","       <use xlink:href=\"#ma8aaa4f60f\" x=\"51.378125\" y=\"61.790117\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n","      </g>\n","     </g>\n","     <g id=\"text_8\">\n","      <!-- $\\mathdefault{10^{0}}$ -->\n","      <g transform=\"translate(26.778125 65.589335)scale(0.1 -0.1)\">\n","       <use xlink:href=\"#DejaVuSans-31\" transform=\"translate(0 0.765625)\"/>\n","       <use xlink:href=\"#DejaVuSans-30\" transform=\"translate(63.623047 0.765625)\"/>\n","       <use xlink:href=\"#DejaVuSans-30\" transform=\"translate(128.203125 39.046875)scale(0.7)\"/>\n","      </g>\n","     </g>\n","    </g>\n","    <g id=\"ytick_3\">\n","     <g id=\"line2d_15\">\n","      <path d=\"M 51.378125 10.999219 \n","L 246.678125 10.999219 \n","\" clip-path=\"url(#pd83c5e35c7)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n","     </g>\n","     <g id=\"line2d_16\">\n","      <g>\n","       <use xlink:href=\"#ma8aaa4f60f\" x=\"51.378125\" y=\"10.999219\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n","      </g>\n","     </g>\n","     <g id=\"text_9\">\n","      <!-- $\\mathdefault{10^{2}}$ -->\n","      <g transform=\"translate(26.778125 14.798437)scale(0.1 -0.1)\">\n","       <use xlink:href=\"#DejaVuSans-31\" transform=\"translate(0 0.765625)\"/>\n","       <use xlink:href=\"#DejaVuSans-30\" transform=\"translate(63.623047 0.765625)\"/>\n","       <use xlink:href=\"#DejaVuSans-32\" transform=\"translate(128.203125 39.046875)scale(0.7)\"/>\n","      </g>\n","     </g>\n","    </g>\n","    <g id=\"text_10\">\n","     <!-- loss -->\n","     <g transform=\"translate(14.798437 86.751829)rotate(-90)scale(0.1 -0.1)\">\n","      <defs>\n","       <path id=\"DejaVuSans-6c\" d=\"M 603 4863 \n","L 1178 4863 \n","L 1178 0 \n","L 603 0 \n","L 603 4863 \n","z\n","\" transform=\"scale(0.015625)\"/>\n","      </defs>\n","      <use xlink:href=\"#DejaVuSans-6c\"/>\n","      <use xlink:href=\"#DejaVuSans-6f\" x=\"27.783203\"/>\n","      <use xlink:href=\"#DejaVuSans-73\" x=\"88.964844\"/>\n","      <use xlink:href=\"#DejaVuSans-73\" x=\"141.064453\"/>\n","     </g>\n","    </g>\n","   </g>\n","   <g id=\"line2d_17\">\n","    <path d=\"M 51.378125 34.334134 \n","L 61.657072 50.369663 \n","L 71.93602 65.62548 \n","L 82.214967 80.348757 \n","L 92.493914 94.697254 \n","L 102.772862 108.615806 \n","L 113.051809 121.476602 \n","L 123.330757 131.489245 \n","L 133.609704 136.728038 \n","L 143.888651 138.225215 \n","L 154.167599 138.387269 \n","L 164.446546 138.194005 \n","L 174.725493 138.189174 \n","L 185.004441 138.179419 \n","L 195.283388 138.096809 \n","L 205.562336 138.364611 \n","L 215.841283 138.475723 \n","L 226.12023 138.682048 \n","L 236.399178 138.846207 \n","L 246.678125 138.866744 \n","\" clip-path=\"url(#pd83c5e35c7)\" style=\"fill: none; stroke: #1f77b4; stroke-width: 1.5; stroke-linecap: square\"/>\n","   </g>\n","   <g id=\"line2d_18\">\n","    <path d=\"M 51.378125 15.321289 \n","L 61.657072 19.257826 \n","L 71.93602 23.181495 \n","L 82.214967 27.1216 \n","L 92.493914 31.078954 \n","L 102.772862 35.050837 \n","L 113.051809 39.032385 \n","L 123.330757 43.02071 \n","L 133.609704 47.012623 \n","L 143.888651 51.005697 \n","L 154.167599 54.99629 \n","L 164.446546 58.980406 \n","L 174.725493 62.952397 \n","L 185.004441 66.90531 \n","L 195.283388 70.827173 \n","L 205.562336 74.703102 \n","L 215.841283 78.512446 \n","L 226.12023 82.224176 \n","L 236.399178 85.803689 \n","L 246.678125 89.204236 \n","\" clip-path=\"url(#pd83c5e35c7)\" style=\"fill: none; stroke-dasharray: 5.55,2.4; stroke-dashoffset: 0; stroke: #bf00bf; stroke-width: 1.5\"/>\n","   </g>\n","   <g id=\"patch_3\">\n","    <path d=\"M 51.378125 145.044016 \n","L 51.378125 9.144016 \n","\" style=\"fill: none; stroke: #000000; stroke-width: 0.8; stroke-linejoin: miter; stroke-linecap: square\"/>\n","   </g>\n","   <g id=\"patch_4\">\n","    <path d=\"M 246.678125 145.044016 \n","L 246.678125 9.144016 \n","\" style=\"fill: none; stroke: #000000; stroke-width: 0.8; stroke-linejoin: miter; stroke-linecap: square\"/>\n","   </g>\n","   <g id=\"patch_5\">\n","    <path d=\"M 51.378125 145.044016 \n","L 246.678125 145.044016 \n","\" style=\"fill: none; stroke: #000000; stroke-width: 0.8; stroke-linejoin: miter; stroke-linecap: square\"/>\n","   </g>\n","   <g id=\"patch_6\">\n","    <path d=\"M 51.378125 9.144016 \n","L 246.678125 9.144016 \n","\" style=\"fill: none; stroke: #000000; stroke-width: 0.8; stroke-linejoin: miter; stroke-linecap: square\"/>\n","   </g>\n","   <g id=\"legend_1\">\n","    <g id=\"patch_7\">\n","     <path d=\"M 184.403125 46.500266 \n","L 239.678125 46.500266 \n","Q 241.678125 46.500266 241.678125 44.500266 \n","L 241.678125 16.144016 \n","Q 241.678125 14.144016 239.678125 14.144016 \n","L 184.403125 14.144016 \n","Q 182.403125 14.144016 182.403125 16.144016 \n","L 182.403125 44.500266 \n","Q 182.403125 46.500266 184.403125 46.500266 \n","z\n","\" style=\"fill: #ffffff; opacity: 0.8; stroke: #cccccc; stroke-linejoin: miter\"/>\n","    </g>\n","    <g id=\"line2d_19\">\n","     <path d=\"M 186.403125 22.242454 \n","L 196.403125 22.242454 \n","L 206.403125 22.242454 \n","\" style=\"fill: none; stroke: #1f77b4; stroke-width: 1.5; stroke-linecap: square\"/>\n","    </g>\n","    <g id=\"text_11\">\n","     <!-- train -->\n","     <g transform=\"translate(214.403125 25.742454)scale(0.1 -0.1)\">\n","      <defs>\n","       <path id=\"DejaVuSans-74\" d=\"M 1172 4494 \n","L 1172 3500 \n","L 2356 3500 \n","L 2356 3053 \n","L 1172 3053 \n","L 1172 1153 \n","Q 1172 725 1289 603 \n","Q 1406 481 1766 481 \n","L 2356 481 \n","L 2356 0 \n","L 1766 0 \n","Q 1100 0 847 248 \n","Q 594 497 594 1153 \n","L 594 3053 \n","L 172 3053 \n","L 172 3500 \n","L 594 3500 \n","L 594 4494 \n","L 1172 4494 \n","z\n","\" transform=\"scale(0.015625)\"/>\n","       <path id=\"DejaVuSans-72\" d=\"M 2631 2963 \n","Q 2534 3019 2420 3045 \n","Q 2306 3072 2169 3072 \n","Q 1681 3072 1420 2755 \n","Q 1159 2438 1159 1844 \n","L 1159 0 \n","L 581 0 \n","L 581 3500 \n","L 1159 3500 \n","L 1159 2956 \n","Q 1341 3275 1631 3429 \n","Q 1922 3584 2338 3584 \n","Q 2397 3584 2469 3576 \n","Q 2541 3569 2628 3553 \n","L 2631 2963 \n","z\n","\" transform=\"scale(0.015625)\"/>\n","       <path id=\"DejaVuSans-61\" d=\"M 2194 1759 \n","Q 1497 1759 1228 1600 \n","Q 959 1441 959 1056 \n","Q 959 750 1161 570 \n","Q 1363 391 1709 391 \n","Q 2188 391 2477 730 \n","Q 2766 1069 2766 1631 \n","L 2766 1759 \n","L 2194 1759 \n","z\n","M 3341 1997 \n","L 3341 0 \n","L 2766 0 \n","L 2766 531 \n","Q 2569 213 2275 61 \n","Q 1981 -91 1556 -91 \n","Q 1019 -91 701 211 \n","Q 384 513 384 1019 \n","Q 384 1609 779 1909 \n","Q 1175 2209 1959 2209 \n","L 2766 2209 \n","L 2766 2266 \n","Q 2766 2663 2505 2880 \n","Q 2244 3097 1772 3097 \n","Q 1472 3097 1187 3025 \n","Q 903 2953 641 2809 \n","L 641 3341 \n","Q 956 3463 1253 3523 \n","Q 1550 3584 1831 3584 \n","Q 2591 3584 2966 3190 \n","Q 3341 2797 3341 1997 \n","z\n","\" transform=\"scale(0.015625)\"/>\n","       <path id=\"DejaVuSans-69\" d=\"M 603 3500 \n","L 1178 3500 \n","L 1178 0 \n","L 603 0 \n","L 603 3500 \n","z\n","M 603 4863 \n","L 1178 4863 \n","L 1178 4134 \n","L 603 4134 \n","L 603 4863 \n","z\n","\" transform=\"scale(0.015625)\"/>\n","       <path id=\"DejaVuSans-6e\" d=\"M 3513 2113 \n","L 3513 0 \n","L 2938 0 \n","L 2938 2094 \n","Q 2938 2591 2744 2837 \n","Q 2550 3084 2163 3084 \n","Q 1697 3084 1428 2787 \n","Q 1159 2491 1159 1978 \n","L 1159 0 \n","L 581 0 \n","L 581 3500 \n","L 1159 3500 \n","L 1159 2956 \n","Q 1366 3272 1645 3428 \n","Q 1925 3584 2291 3584 \n","Q 2894 3584 3203 3211 \n","Q 3513 2838 3513 2113 \n","z\n","\" transform=\"scale(0.015625)\"/>\n","      </defs>\n","      <use xlink:href=\"#DejaVuSans-74\"/>\n","      <use xlink:href=\"#DejaVuSans-72\" x=\"39.208984\"/>\n","      <use xlink:href=\"#DejaVuSans-61\" x=\"80.322266\"/>\n","      <use xlink:href=\"#DejaVuSans-69\" x=\"141.601562\"/>\n","      <use xlink:href=\"#DejaVuSans-6e\" x=\"169.384766\"/>\n","     </g>\n","    </g>\n","    <g id=\"line2d_20\">\n","     <path d=\"M 186.403125 36.920579 \n","L 196.403125 36.920579 \n","L 206.403125 36.920579 \n","\" style=\"fill: none; stroke-dasharray: 5.55,2.4; stroke-dashoffset: 0; stroke: #bf00bf; stroke-width: 1.5\"/>\n","    </g>\n","    <g id=\"text_12\">\n","     <!-- test -->\n","     <g transform=\"translate(214.403125 40.420579)scale(0.1 -0.1)\">\n","      <use xlink:href=\"#DejaVuSans-74\"/>\n","      <use xlink:href=\"#DejaVuSans-65\" x=\"39.208984\"/>\n","      <use xlink:href=\"#DejaVuSans-73\" x=\"100.732422\"/>\n","      <use xlink:href=\"#DejaVuSans-74\" x=\"152.832031\"/>\n","     </g>\n","    </g>\n","   </g>\n","  </g>\n"," </g>\n"," <defs>\n","  <clipPath id=\"pd83c5e35c7\">\n","   <rect x=\"51.378125\" y=\"9.144016\" width=\"195.3\" height=\"135.9\"/>\n","  </clipPath>\n"," </defs>\n","</svg>\n"],"text/plain":["<Figure size 252x180 with 1 Axes>"]},"metadata":{"needs_background":"light"},"output_type":"display_data"}],"source":["train(lambd=3)"],"id":"SxRzn_szhB9D"},{"cell_type":"markdown","metadata":{"origin_pos":22,"id":"C272ivC1hB9D"},"source":["## [**简洁实现**]\n","\n","由于权重衰减在神经网络优化中很常用，\n","深度学习框架为了便于我们使用权重衰减，\n","将权重衰减集成到优化算法中，以便与任何损失函数结合使用。\n","此外，这种集成还有计算上的好处，\n","允许在不增加任何额外的计算开销的情况下向算法中添加权重衰减。\n","由于更新的权重衰减部分仅依赖于每个参数的当前值，\n","因此优化器必须至少接触每个参数一次。\n"],"id":"C272ivC1hB9D"},{"cell_type":"markdown","metadata":{"origin_pos":24,"tab":["pytorch"],"id":"QPCYqyCChB9D"},"source":["在下面的代码中，我们在实例化优化器时直接通过`weight_decay`指定weight decay超参数。\n","默认情况下，PyTorch同时衰减权重和偏移。\n","这里我们只为权重设置了`weight_decay`，所以偏置参数$b$不会衰减。\n"],"id":"QPCYqyCChB9D"},{"cell_type":"code","execution_count":null,"metadata":{"execution":{"iopub.execute_input":"2022-07-31T02:23:01.983489Z","iopub.status.busy":"2022-07-31T02:23:01.982884Z","iopub.status.idle":"2022-07-31T02:23:01.991713Z","shell.execute_reply":"2022-07-31T02:23:01.990937Z"},"origin_pos":27,"tab":["pytorch"],"id":"5YEfkdJGhB9E"},"outputs":[],"source":["def train_concise(wd):\n","    net = nn.Sequential(nn.Linear(num_inputs, 1))\n","    for param in net.parameters():\n","        param.data.normal_()\n","    loss = nn.MSELoss(reduction='none')\n","    num_epochs, lr = 100, 0.003\n","    # 偏置参数没有衰减\n","    trainer = torch.optim.SGD([\n","        {\"params\":net[0].weight,'weight_decay': wd},\n","        {\"params\":net[0].bias}], lr=lr)\n","    animator = d2l.Animator(xlabel='epochs', ylabel='loss', yscale='log',\n","                            xlim=[5, num_epochs], legend=['train', 'test'])\n","    for epoch in range(num_epochs):\n","        for X, y in train_iter:\n","            trainer.zero_grad()\n","            l = loss(net(X), y)\n","            l.mean().backward()\n","            trainer.step()\n","        if (epoch + 1) % 5 == 0:\n","            animator.add(epoch + 1,\n","                         (d2l.evaluate_loss(net, train_iter, loss),\n","                          d2l.evaluate_loss(net, test_iter, loss)))\n","    print('w的L2范数：', net[0].weight.norm().item())"],"id":"5YEfkdJGhB9E"},{"cell_type":"markdown","metadata":{"origin_pos":29,"id":"sDq1gm_fhB9E"},"source":["[**这些图看起来和我们从零开始实现权重衰减时的图相同**]。\n","然而，它们运行得更快，更容易实现。\n","对于更复杂的问题，这一好处将变得更加明显。\n"],"id":"sDq1gm_fhB9E"},{"cell_type":"code","execution_count":null,"metadata":{"execution":{"iopub.execute_input":"2022-07-31T02:23:01.995096Z","iopub.status.busy":"2022-07-31T02:23:01.994625Z","iopub.status.idle":"2022-07-31T02:23:07.965364Z","shell.execute_reply":"2022-07-31T02:23:07.964719Z"},"origin_pos":30,"tab":["pytorch"],"outputId":"20639d14-5066-4be4-ba3e-3518fc97ee92","id":"oxh8fB35hB9E"},"outputs":[{"name":"stdout","output_type":"stream","text":["w的L2范数： 14.026016235351562\n"]},{"data":{"image/svg+xml":["<?xml version=\"1.0\" encoding=\"utf-8\" standalone=\"no\"?>\n","<!DOCTYPE svg PUBLIC \"-//W3C//DTD SVG 1.1//EN\"\n","  \"http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd\">\n","<svg xmlns:xlink=\"http://www.w3.org/1999/xlink\" width=\"263.421875pt\" height=\"180.65625pt\" viewBox=\"0 0 263.421875 180.65625\" xmlns=\"http://www.w3.org/2000/svg\" version=\"1.1\">\n"," <metadata>\n","  <rdf:RDF xmlns:dc=\"http://purl.org/dc/elements/1.1/\" xmlns:cc=\"http://creativecommons.org/ns#\" xmlns:rdf=\"http://www.w3.org/1999/02/22-rdf-syntax-ns#\">\n","   <cc:Work>\n","    <dc:type rdf:resource=\"http://purl.org/dc/dcmitype/StillImage\"/>\n","    <dc:date>2022-07-31T02:23:07.884250</dc:date>\n","    <dc:format>image/svg+xml</dc:format>\n","    <dc:creator>\n","     <cc:Agent>\n","      <dc:title>Matplotlib v3.5.1, https://matplotlib.org/</dc:title>\n","     </cc:Agent>\n","    </dc:creator>\n","   </cc:Work>\n","  </rdf:RDF>\n"," </metadata>\n"," <defs>\n","  <style type=\"text/css\">*{stroke-linejoin: round; stroke-linecap: butt}</style>\n"," </defs>\n"," <g id=\"figure_1\">\n","  <g id=\"patch_1\">\n","   <path d=\"M 0 180.65625 \n","L 263.421875 180.65625 \n","L 263.421875 0 \n","L 0 0 \n","L 0 180.65625 \n","z\n","\" style=\"fill: none\"/>\n","  </g>\n","  <g id=\"axes_1\">\n","   <g id=\"patch_2\">\n","    <path d=\"M 51.378125 143.1 \n","L 246.678125 143.1 \n","L 246.678125 7.2 \n","L 51.378125 7.2 \n","z\n","\" style=\"fill: #ffffff\"/>\n","   </g>\n","   <g id=\"matplotlib.axis_1\">\n","    <g id=\"xtick_1\">\n","     <g id=\"line2d_1\">\n","      <path d=\"M 82.214967 143.1 \n","L 82.214967 7.2 \n","\" clip-path=\"url(#p5ed042e645)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n","     </g>\n","     <g id=\"line2d_2\">\n","      <defs>\n","       <path id=\"me20a397f43\" d=\"M 0 0 \n","L 0 3.5 \n","\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n","      </defs>\n","      <g>\n","       <use xlink:href=\"#me20a397f43\" x=\"82.214967\" y=\"143.1\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n","      </g>\n","     </g>\n","     <g id=\"text_1\">\n","      <!-- 20 -->\n","      <g transform=\"translate(75.852467 157.698438)scale(0.1 -0.1)\">\n","       <defs>\n","        <path id=\"DejaVuSans-32\" d=\"M 1228 531 \n","L 3431 531 \n","L 3431 0 \n","L 469 0 \n","L 469 531 \n","Q 828 903 1448 1529 \n","Q 2069 2156 2228 2338 \n","Q 2531 2678 2651 2914 \n","Q 2772 3150 2772 3378 \n","Q 2772 3750 2511 3984 \n","Q 2250 4219 1831 4219 \n","Q 1534 4219 1204 4116 \n","Q 875 4013 500 3803 \n","L 500 4441 \n","Q 881 4594 1212 4672 \n","Q 1544 4750 1819 4750 \n","Q 2544 4750 2975 4387 \n","Q 3406 4025 3406 3419 \n","Q 3406 3131 3298 2873 \n","Q 3191 2616 2906 2266 \n","Q 2828 2175 2409 1742 \n","Q 1991 1309 1228 531 \n","z\n","\" transform=\"scale(0.015625)\"/>\n","        <path id=\"DejaVuSans-30\" d=\"M 2034 4250 \n","Q 1547 4250 1301 3770 \n","Q 1056 3291 1056 2328 \n","Q 1056 1369 1301 889 \n","Q 1547 409 2034 409 \n","Q 2525 409 2770 889 \n","Q 3016 1369 3016 2328 \n","Q 3016 3291 2770 3770 \n","Q 2525 4250 2034 4250 \n","z\n","M 2034 4750 \n","Q 2819 4750 3233 4129 \n","Q 3647 3509 3647 2328 \n","Q 3647 1150 3233 529 \n","Q 2819 -91 2034 -91 \n","Q 1250 -91 836 529 \n","Q 422 1150 422 2328 \n","Q 422 3509 836 4129 \n","Q 1250 4750 2034 4750 \n","z\n","\" transform=\"scale(0.015625)\"/>\n","       </defs>\n","       <use xlink:href=\"#DejaVuSans-32\"/>\n","       <use xlink:href=\"#DejaVuSans-30\" x=\"63.623047\"/>\n","      </g>\n","     </g>\n","    </g>\n","    <g id=\"xtick_2\">\n","     <g id=\"line2d_3\">\n","      <path d=\"M 123.330757 143.1 \n","L 123.330757 7.2 \n","\" clip-path=\"url(#p5ed042e645)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n","     </g>\n","     <g id=\"line2d_4\">\n","      <g>\n","       <use xlink:href=\"#me20a397f43\" x=\"123.330757\" y=\"143.1\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n","      </g>\n","     </g>\n","     <g id=\"text_2\">\n","      <!-- 40 -->\n","      <g transform=\"translate(116.968257 157.698438)scale(0.1 -0.1)\">\n","       <defs>\n","        <path id=\"DejaVuSans-34\" d=\"M 2419 4116 \n","L 825 1625 \n","L 2419 1625 \n","L 2419 4116 \n","z\n","M 2253 4666 \n","L 3047 4666 \n","L 3047 1625 \n","L 3713 1625 \n","L 3713 1100 \n","L 3047 1100 \n","L 3047 0 \n","L 2419 0 \n","L 2419 1100 \n","L 313 1100 \n","L 313 1709 \n","L 2253 4666 \n","z\n","\" transform=\"scale(0.015625)\"/>\n","       </defs>\n","       <use xlink:href=\"#DejaVuSans-34\"/>\n","       <use xlink:href=\"#DejaVuSans-30\" x=\"63.623047\"/>\n","      </g>\n","     </g>\n","    </g>\n","    <g id=\"xtick_3\">\n","     <g id=\"line2d_5\">\n","      <path d=\"M 164.446546 143.1 \n","L 164.446546 7.2 \n","\" clip-path=\"url(#p5ed042e645)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n","     </g>\n","     <g id=\"line2d_6\">\n","      <g>\n","       <use xlink:href=\"#me20a397f43\" x=\"164.446546\" y=\"143.1\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n","      </g>\n","     </g>\n","     <g id=\"text_3\">\n","      <!-- 60 -->\n","      <g transform=\"translate(158.084046 157.698438)scale(0.1 -0.1)\">\n","       <defs>\n","        <path id=\"DejaVuSans-36\" d=\"M 2113 2584 \n","Q 1688 2584 1439 2293 \n","Q 1191 2003 1191 1497 \n","Q 1191 994 1439 701 \n","Q 1688 409 2113 409 \n","Q 2538 409 2786 701 \n","Q 3034 994 3034 1497 \n","Q 3034 2003 2786 2293 \n","Q 2538 2584 2113 2584 \n","z\n","M 3366 4563 \n","L 3366 3988 \n","Q 3128 4100 2886 4159 \n","Q 2644 4219 2406 4219 \n","Q 1781 4219 1451 3797 \n","Q 1122 3375 1075 2522 \n","Q 1259 2794 1537 2939 \n","Q 1816 3084 2150 3084 \n","Q 2853 3084 3261 2657 \n","Q 3669 2231 3669 1497 \n","Q 3669 778 3244 343 \n","Q 2819 -91 2113 -91 \n","Q 1303 -91 875 529 \n","Q 447 1150 447 2328 \n","Q 447 3434 972 4092 \n","Q 1497 4750 2381 4750 \n","Q 2619 4750 2861 4703 \n","Q 3103 4656 3366 4563 \n","z\n","\" transform=\"scale(0.015625)\"/>\n","       </defs>\n","       <use xlink:href=\"#DejaVuSans-36\"/>\n","       <use xlink:href=\"#DejaVuSans-30\" x=\"63.623047\"/>\n","      </g>\n","     </g>\n","    </g>\n","    <g id=\"xtick_4\">\n","     <g id=\"line2d_7\">\n","      <path d=\"M 205.562336 143.1 \n","L 205.562336 7.2 \n","\" clip-path=\"url(#p5ed042e645)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n","     </g>\n","     <g id=\"line2d_8\">\n","      <g>\n","       <use xlink:href=\"#me20a397f43\" x=\"205.562336\" y=\"143.1\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n","      </g>\n","     </g>\n","     <g id=\"text_4\">\n","      <!-- 80 -->\n","      <g transform=\"translate(199.199836 157.698438)scale(0.1 -0.1)\">\n","       <defs>\n","        <path id=\"DejaVuSans-38\" d=\"M 2034 2216 \n","Q 1584 2216 1326 1975 \n","Q 1069 1734 1069 1313 \n","Q 1069 891 1326 650 \n","Q 1584 409 2034 409 \n","Q 2484 409 2743 651 \n","Q 3003 894 3003 1313 \n","Q 3003 1734 2745 1975 \n","Q 2488 2216 2034 2216 \n","z\n","M 1403 2484 \n","Q 997 2584 770 2862 \n","Q 544 3141 544 3541 \n","Q 544 4100 942 4425 \n","Q 1341 4750 2034 4750 \n","Q 2731 4750 3128 4425 \n","Q 3525 4100 3525 3541 \n","Q 3525 3141 3298 2862 \n","Q 3072 2584 2669 2484 \n","Q 3125 2378 3379 2068 \n","Q 3634 1759 3634 1313 \n","Q 3634 634 3220 271 \n","Q 2806 -91 2034 -91 \n","Q 1263 -91 848 271 \n","Q 434 634 434 1313 \n","Q 434 1759 690 2068 \n","Q 947 2378 1403 2484 \n","z\n","M 1172 3481 \n","Q 1172 3119 1398 2916 \n","Q 1625 2713 2034 2713 \n","Q 2441 2713 2670 2916 \n","Q 2900 3119 2900 3481 \n","Q 2900 3844 2670 4047 \n","Q 2441 4250 2034 4250 \n","Q 1625 4250 1398 4047 \n","Q 1172 3844 1172 3481 \n","z\n","\" transform=\"scale(0.015625)\"/>\n","       </defs>\n","       <use xlink:href=\"#DejaVuSans-38\"/>\n","       <use xlink:href=\"#DejaVuSans-30\" x=\"63.623047\"/>\n","      </g>\n","     </g>\n","    </g>\n","    <g id=\"xtick_5\">\n","     <g id=\"line2d_9\">\n","      <path d=\"M 246.678125 143.1 \n","L 246.678125 7.2 \n","\" clip-path=\"url(#p5ed042e645)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n","     </g>\n","     <g id=\"line2d_10\">\n","      <g>\n","       <use xlink:href=\"#me20a397f43\" x=\"246.678125\" y=\"143.1\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n","      </g>\n","     </g>\n","     <g id=\"text_5\">\n","      <!-- 100 -->\n","      <g transform=\"translate(237.134375 157.698438)scale(0.1 -0.1)\">\n","       <defs>\n","        <path id=\"DejaVuSans-31\" d=\"M 794 531 \n","L 1825 531 \n","L 1825 4091 \n","L 703 3866 \n","L 703 4441 \n","L 1819 4666 \n","L 2450 4666 \n","L 2450 531 \n","L 3481 531 \n","L 3481 0 \n","L 794 0 \n","L 794 531 \n","z\n","\" transform=\"scale(0.015625)\"/>\n","       </defs>\n","       <use xlink:href=\"#DejaVuSans-31\"/>\n","       <use xlink:href=\"#DejaVuSans-30\" x=\"63.623047\"/>\n","       <use xlink:href=\"#DejaVuSans-30\" x=\"127.246094\"/>\n","      </g>\n","     </g>\n","    </g>\n","    <g id=\"text_6\">\n","     <!-- epochs -->\n","     <g transform=\"translate(131.195312 171.376563)scale(0.1 -0.1)\">\n","      <defs>\n","       <path id=\"DejaVuSans-65\" d=\"M 3597 1894 \n","L 3597 1613 \n","L 953 1613 \n","Q 991 1019 1311 708 \n","Q 1631 397 2203 397 \n","Q 2534 397 2845 478 \n","Q 3156 559 3463 722 \n","L 3463 178 \n","Q 3153 47 2828 -22 \n","Q 2503 -91 2169 -91 \n","Q 1331 -91 842 396 \n","Q 353 884 353 1716 \n","Q 353 2575 817 3079 \n","Q 1281 3584 2069 3584 \n","Q 2775 3584 3186 3129 \n","Q 3597 2675 3597 1894 \n","z\n","M 3022 2063 \n","Q 3016 2534 2758 2815 \n","Q 2500 3097 2075 3097 \n","Q 1594 3097 1305 2825 \n","Q 1016 2553 972 2059 \n","L 3022 2063 \n","z\n","\" transform=\"scale(0.015625)\"/>\n","       <path id=\"DejaVuSans-70\" d=\"M 1159 525 \n","L 1159 -1331 \n","L 581 -1331 \n","L 581 3500 \n","L 1159 3500 \n","L 1159 2969 \n","Q 1341 3281 1617 3432 \n","Q 1894 3584 2278 3584 \n","Q 2916 3584 3314 3078 \n","Q 3713 2572 3713 1747 \n","Q 3713 922 3314 415 \n","Q 2916 -91 2278 -91 \n","Q 1894 -91 1617 61 \n","Q 1341 213 1159 525 \n","z\n","M 3116 1747 \n","Q 3116 2381 2855 2742 \n","Q 2594 3103 2138 3103 \n","Q 1681 3103 1420 2742 \n","Q 1159 2381 1159 1747 \n","Q 1159 1113 1420 752 \n","Q 1681 391 2138 391 \n","Q 2594 391 2855 752 \n","Q 3116 1113 3116 1747 \n","z\n","\" transform=\"scale(0.015625)\"/>\n","       <path id=\"DejaVuSans-6f\" d=\"M 1959 3097 \n","Q 1497 3097 1228 2736 \n","Q 959 2375 959 1747 \n","Q 959 1119 1226 758 \n","Q 1494 397 1959 397 \n","Q 2419 397 2687 759 \n","Q 2956 1122 2956 1747 \n","Q 2956 2369 2687 2733 \n","Q 2419 3097 1959 3097 \n","z\n","M 1959 3584 \n","Q 2709 3584 3137 3096 \n","Q 3566 2609 3566 1747 \n","Q 3566 888 3137 398 \n","Q 2709 -91 1959 -91 \n","Q 1206 -91 779 398 \n","Q 353 888 353 1747 \n","Q 353 2609 779 3096 \n","Q 1206 3584 1959 3584 \n","z\n","\" transform=\"scale(0.015625)\"/>\n","       <path id=\"DejaVuSans-63\" d=\"M 3122 3366 \n","L 3122 2828 \n","Q 2878 2963 2633 3030 \n","Q 2388 3097 2138 3097 \n","Q 1578 3097 1268 2742 \n","Q 959 2388 959 1747 \n","Q 959 1106 1268 751 \n","Q 1578 397 2138 397 \n","Q 2388 397 2633 464 \n","Q 2878 531 3122 666 \n","L 3122 134 \n","Q 2881 22 2623 -34 \n","Q 2366 -91 2075 -91 \n","Q 1284 -91 818 406 \n","Q 353 903 353 1747 \n","Q 353 2603 823 3093 \n","Q 1294 3584 2113 3584 \n","Q 2378 3584 2631 3529 \n","Q 2884 3475 3122 3366 \n","z\n","\" transform=\"scale(0.015625)\"/>\n","       <path id=\"DejaVuSans-68\" d=\"M 3513 2113 \n","L 3513 0 \n","L 2938 0 \n","L 2938 2094 \n","Q 2938 2591 2744 2837 \n","Q 2550 3084 2163 3084 \n","Q 1697 3084 1428 2787 \n","Q 1159 2491 1159 1978 \n","L 1159 0 \n","L 581 0 \n","L 581 4863 \n","L 1159 4863 \n","L 1159 2956 \n","Q 1366 3272 1645 3428 \n","Q 1925 3584 2291 3584 \n","Q 2894 3584 3203 3211 \n","Q 3513 2838 3513 2113 \n","z\n","\" transform=\"scale(0.015625)\"/>\n","       <path id=\"DejaVuSans-73\" d=\"M 2834 3397 \n","L 2834 2853 \n","Q 2591 2978 2328 3040 \n","Q 2066 3103 1784 3103 \n","Q 1356 3103 1142 2972 \n","Q 928 2841 928 2578 \n","Q 928 2378 1081 2264 \n","Q 1234 2150 1697 2047 \n","L 1894 2003 \n","Q 2506 1872 2764 1633 \n","Q 3022 1394 3022 966 \n","Q 3022 478 2636 193 \n","Q 2250 -91 1575 -91 \n","Q 1294 -91 989 -36 \n","Q 684 19 347 128 \n","L 347 722 \n","Q 666 556 975 473 \n","Q 1284 391 1588 391 \n","Q 1994 391 2212 530 \n","Q 2431 669 2431 922 \n","Q 2431 1156 2273 1281 \n","Q 2116 1406 1581 1522 \n","L 1381 1569 \n","Q 847 1681 609 1914 \n","Q 372 2147 372 2553 \n","Q 372 3047 722 3315 \n","Q 1072 3584 1716 3584 \n","Q 2034 3584 2315 3537 \n","Q 2597 3491 2834 3397 \n","z\n","\" transform=\"scale(0.015625)\"/>\n","      </defs>\n","      <use xlink:href=\"#DejaVuSans-65\"/>\n","      <use xlink:href=\"#DejaVuSans-70\" x=\"61.523438\"/>\n","      <use xlink:href=\"#DejaVuSans-6f\" x=\"125\"/>\n","      <use xlink:href=\"#DejaVuSans-63\" x=\"186.181641\"/>\n","      <use xlink:href=\"#DejaVuSans-68\" x=\"241.162109\"/>\n","      <use xlink:href=\"#DejaVuSans-73\" x=\"304.541016\"/>\n","     </g>\n","    </g>\n","   </g>\n","   <g id=\"matplotlib.axis_2\">\n","    <g id=\"ytick_1\">\n","     <g id=\"line2d_11\">\n","      <path d=\"M 51.378125 119.277003 \n","L 246.678125 119.277003 \n","\" clip-path=\"url(#p5ed042e645)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n","     </g>\n","     <g id=\"line2d_12\">\n","      <defs>\n","       <path id=\"mb887f30291\" d=\"M 0 0 \n","L -3.5 0 \n","\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n","      </defs>\n","      <g>\n","       <use xlink:href=\"#mb887f30291\" x=\"51.378125\" y=\"119.277003\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n","      </g>\n","     </g>\n","     <g id=\"text_7\">\n","      <!-- $\\mathdefault{10^{-9}}$ -->\n","      <g transform=\"translate(20.878125 123.076222)scale(0.1 -0.1)\">\n","       <defs>\n","        <path id=\"DejaVuSans-2212\" d=\"M 678 2272 \n","L 4684 2272 \n","L 4684 1741 \n","L 678 1741 \n","L 678 2272 \n","z\n","\" transform=\"scale(0.015625)\"/>\n","        <path id=\"DejaVuSans-39\" d=\"M 703 97 \n","L 703 672 \n","Q 941 559 1184 500 \n","Q 1428 441 1663 441 \n","Q 2288 441 2617 861 \n","Q 2947 1281 2994 2138 \n","Q 2813 1869 2534 1725 \n","Q 2256 1581 1919 1581 \n","Q 1219 1581 811 2004 \n","Q 403 2428 403 3163 \n","Q 403 3881 828 4315 \n","Q 1253 4750 1959 4750 \n","Q 2769 4750 3195 4129 \n","Q 3622 3509 3622 2328 \n","Q 3622 1225 3098 567 \n","Q 2575 -91 1691 -91 \n","Q 1453 -91 1209 -44 \n","Q 966 3 703 97 \n","z\n","M 1959 2075 \n","Q 2384 2075 2632 2365 \n","Q 2881 2656 2881 3163 \n","Q 2881 3666 2632 3958 \n","Q 2384 4250 1959 4250 \n","Q 1534 4250 1286 3958 \n","Q 1038 3666 1038 3163 \n","Q 1038 2656 1286 2365 \n","Q 1534 2075 1959 2075 \n","z\n","\" transform=\"scale(0.015625)\"/>\n","       </defs>\n","       <use xlink:href=\"#DejaVuSans-31\" transform=\"translate(0 0.765625)\"/>\n","       <use xlink:href=\"#DejaVuSans-30\" transform=\"translate(63.623047 0.765625)\"/>\n","       <use xlink:href=\"#DejaVuSans-2212\" transform=\"translate(128.203125 39.046875)scale(0.7)\"/>\n","       <use xlink:href=\"#DejaVuSans-39\" transform=\"translate(186.855469 39.046875)scale(0.7)\"/>\n","      </g>\n","     </g>\n","    </g>\n","    <g id=\"ytick_2\">\n","     <g id=\"line2d_13\">\n","      <path d=\"M 51.378125 91.178407 \n","L 246.678125 91.178407 \n","\" clip-path=\"url(#p5ed042e645)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n","     </g>\n","     <g id=\"line2d_14\">\n","      <g>\n","       <use xlink:href=\"#mb887f30291\" x=\"51.378125\" y=\"91.178407\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n","      </g>\n","     </g>\n","     <g id=\"text_8\">\n","      <!-- $\\mathdefault{10^{-6}}$ -->\n","      <g transform=\"translate(20.878125 94.977626)scale(0.1 -0.1)\">\n","       <use xlink:href=\"#DejaVuSans-31\" transform=\"translate(0 0.765625)\"/>\n","       <use xlink:href=\"#DejaVuSans-30\" transform=\"translate(63.623047 0.765625)\"/>\n","       <use xlink:href=\"#DejaVuSans-2212\" transform=\"translate(128.203125 39.046875)scale(0.7)\"/>\n","       <use xlink:href=\"#DejaVuSans-36\" transform=\"translate(186.855469 39.046875)scale(0.7)\"/>\n","      </g>\n","     </g>\n","    </g>\n","    <g id=\"ytick_3\">\n","     <g id=\"line2d_15\">\n","      <path d=\"M 51.378125 63.079811 \n","L 246.678125 63.079811 \n","\" clip-path=\"url(#p5ed042e645)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n","     </g>\n","     <g id=\"line2d_16\">\n","      <g>\n","       <use xlink:href=\"#mb887f30291\" x=\"51.378125\" y=\"63.079811\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n","      </g>\n","     </g>\n","     <g id=\"text_9\">\n","      <!-- $\\mathdefault{10^{-3}}$ -->\n","      <g transform=\"translate(20.878125 66.87903)scale(0.1 -0.1)\">\n","       <defs>\n","        <path id=\"DejaVuSans-33\" d=\"M 2597 2516 \n","Q 3050 2419 3304 2112 \n","Q 3559 1806 3559 1356 \n","Q 3559 666 3084 287 \n","Q 2609 -91 1734 -91 \n","Q 1441 -91 1130 -33 \n","Q 819 25 488 141 \n","L 488 750 \n","Q 750 597 1062 519 \n","Q 1375 441 1716 441 \n","Q 2309 441 2620 675 \n","Q 2931 909 2931 1356 \n","Q 2931 1769 2642 2001 \n","Q 2353 2234 1838 2234 \n","L 1294 2234 \n","L 1294 2753 \n","L 1863 2753 \n","Q 2328 2753 2575 2939 \n","Q 2822 3125 2822 3475 \n","Q 2822 3834 2567 4026 \n","Q 2313 4219 1838 4219 \n","Q 1578 4219 1281 4162 \n","Q 984 4106 628 3988 \n","L 628 4550 \n","Q 988 4650 1302 4700 \n","Q 1616 4750 1894 4750 \n","Q 2613 4750 3031 4423 \n","Q 3450 4097 3450 3541 \n","Q 3450 3153 3228 2886 \n","Q 3006 2619 2597 2516 \n","z\n","\" transform=\"scale(0.015625)\"/>\n","       </defs>\n","       <use xlink:href=\"#DejaVuSans-31\" transform=\"translate(0 0.765625)\"/>\n","       <use xlink:href=\"#DejaVuSans-30\" transform=\"translate(63.623047 0.765625)\"/>\n","       <use xlink:href=\"#DejaVuSans-2212\" transform=\"translate(128.203125 39.046875)scale(0.7)\"/>\n","       <use xlink:href=\"#DejaVuSans-33\" transform=\"translate(186.855469 39.046875)scale(0.7)\"/>\n","      </g>\n","     </g>\n","    </g>\n","    <g id=\"ytick_4\">\n","     <g id=\"line2d_17\">\n","      <path d=\"M 51.378125 34.981215 \n","L 246.678125 34.981215 \n","\" clip-path=\"url(#p5ed042e645)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n","     </g>\n","     <g id=\"line2d_18\">\n","      <g>\n","       <use xlink:href=\"#mb887f30291\" x=\"51.378125\" y=\"34.981215\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n","      </g>\n","     </g>\n","     <g id=\"text_10\">\n","      <!-- $\\mathdefault{10^{0}}$ -->\n","      <g transform=\"translate(26.778125 38.780434)scale(0.1 -0.1)\">\n","       <use xlink:href=\"#DejaVuSans-31\" transform=\"translate(0 0.765625)\"/>\n","       <use xlink:href=\"#DejaVuSans-30\" transform=\"translate(63.623047 0.765625)\"/>\n","       <use xlink:href=\"#DejaVuSans-30\" transform=\"translate(128.203125 39.046875)scale(0.7)\"/>\n","      </g>\n","     </g>\n","    </g>\n","    <g id=\"text_11\">\n","     <!-- loss -->\n","     <g transform=\"translate(14.798437 84.807812)rotate(-90)scale(0.1 -0.1)\">\n","      <defs>\n","       <path id=\"DejaVuSans-6c\" d=\"M 603 4863 \n","L 1178 4863 \n","L 1178 0 \n","L 603 0 \n","L 603 4863 \n","z\n","\" transform=\"scale(0.015625)\"/>\n","      </defs>\n","      <use xlink:href=\"#DejaVuSans-6c\"/>\n","      <use xlink:href=\"#DejaVuSans-6f\" x=\"27.783203\"/>\n","      <use xlink:href=\"#DejaVuSans-73\" x=\"88.964844\"/>\n","      <use xlink:href=\"#DejaVuSans-73\" x=\"141.064453\"/>\n","     </g>\n","    </g>\n","   </g>\n","   <g id=\"line2d_19\">\n","    <path d=\"M 51.378125 24.168093 \n","L 61.657072 33.711551 \n","L 71.93602 41.915027 \n","L 82.214967 49.524061 \n","L 92.493914 56.738217 \n","L 102.772862 63.769639 \n","L 113.051809 70.64042 \n","L 123.330757 77.402652 \n","L 133.609704 84.085355 \n","L 143.888651 90.66234 \n","L 154.167599 97.107542 \n","L 164.446546 103.43541 \n","L 174.725493 109.653939 \n","L 185.004441 115.661614 \n","L 195.283388 121.408282 \n","L 205.562336 126.776031 \n","L 215.841283 130.811171 \n","L 226.12023 133.648442 \n","L 236.399178 135.602606 \n","L 246.678125 136.922727 \n","\" clip-path=\"url(#p5ed042e645)\" style=\"fill: none; stroke: #1f77b4; stroke-width: 1.5; stroke-linecap: square\"/>\n","   </g>\n","   <g id=\"line2d_20\">\n","    <path d=\"M 51.378125 13.377273 \n","L 61.657072 13.425898 \n","L 71.93602 13.447246 \n","L 82.214967 13.456447 \n","L 92.493914 13.460139 \n","L 102.772862 13.461661 \n","L 113.051809 13.462259 \n","L 123.330757 13.462501 \n","L 133.609704 13.462598 \n","L 143.888651 13.462637 \n","L 154.167599 13.462652 \n","L 164.446546 13.462658 \n","L 174.725493 13.462661 \n","L 185.004441 13.462662 \n","L 195.283388 13.462662 \n","L 205.562336 13.462662 \n","L 215.841283 13.462662 \n","L 226.12023 13.462662 \n","L 236.399178 13.462662 \n","L 246.678125 13.462662 \n","\" clip-path=\"url(#p5ed042e645)\" style=\"fill: none; stroke-dasharray: 5.55,2.4; stroke-dashoffset: 0; stroke: #bf00bf; stroke-width: 1.5\"/>\n","   </g>\n","   <g id=\"patch_3\">\n","    <path d=\"M 51.378125 143.1 \n","L 51.378125 7.2 \n","\" style=\"fill: none; stroke: #000000; stroke-width: 0.8; stroke-linejoin: miter; stroke-linecap: square\"/>\n","   </g>\n","   <g id=\"patch_4\">\n","    <path d=\"M 246.678125 143.1 \n","L 246.678125 7.2 \n","\" style=\"fill: none; stroke: #000000; stroke-width: 0.8; stroke-linejoin: miter; stroke-linecap: square\"/>\n","   </g>\n","   <g id=\"patch_5\">\n","    <path d=\"M 51.378125 143.1 \n","L 246.678125 143.1 \n","\" style=\"fill: none; stroke: #000000; stroke-width: 0.8; stroke-linejoin: miter; stroke-linecap: square\"/>\n","   </g>\n","   <g id=\"patch_6\">\n","    <path d=\"M 51.378125 7.2 \n","L 246.678125 7.2 \n","\" style=\"fill: none; stroke: #000000; stroke-width: 0.8; stroke-linejoin: miter; stroke-linecap: square\"/>\n","   </g>\n","   <g id=\"legend_1\">\n","    <g id=\"patch_7\">\n","     <path d=\"M 58.378125 138.1 \n","L 113.653125 138.1 \n","Q 115.653125 138.1 115.653125 136.1 \n","L 115.653125 107.74375 \n","Q 115.653125 105.74375 113.653125 105.74375 \n","L 58.378125 105.74375 \n","Q 56.378125 105.74375 56.378125 107.74375 \n","L 56.378125 136.1 \n","Q 56.378125 138.1 58.378125 138.1 \n","z\n","\" style=\"fill: #ffffff; opacity: 0.8; stroke: #cccccc; stroke-linejoin: miter\"/>\n","    </g>\n","    <g id=\"line2d_21\">\n","     <path d=\"M 60.378125 113.842187 \n","L 70.378125 113.842187 \n","L 80.378125 113.842187 \n","\" style=\"fill: none; stroke: #1f77b4; stroke-width: 1.5; stroke-linecap: square\"/>\n","    </g>\n","    <g id=\"text_12\">\n","     <!-- train -->\n","     <g transform=\"translate(88.378125 117.342187)scale(0.1 -0.1)\">\n","      <defs>\n","       <path id=\"DejaVuSans-74\" d=\"M 1172 4494 \n","L 1172 3500 \n","L 2356 3500 \n","L 2356 3053 \n","L 1172 3053 \n","L 1172 1153 \n","Q 1172 725 1289 603 \n","Q 1406 481 1766 481 \n","L 2356 481 \n","L 2356 0 \n","L 1766 0 \n","Q 1100 0 847 248 \n","Q 594 497 594 1153 \n","L 594 3053 \n","L 172 3053 \n","L 172 3500 \n","L 594 3500 \n","L 594 4494 \n","L 1172 4494 \n","z\n","\" transform=\"scale(0.015625)\"/>\n","       <path id=\"DejaVuSans-72\" d=\"M 2631 2963 \n","Q 2534 3019 2420 3045 \n","Q 2306 3072 2169 3072 \n","Q 1681 3072 1420 2755 \n","Q 1159 2438 1159 1844 \n","L 1159 0 \n","L 581 0 \n","L 581 3500 \n","L 1159 3500 \n","L 1159 2956 \n","Q 1341 3275 1631 3429 \n","Q 1922 3584 2338 3584 \n","Q 2397 3584 2469 3576 \n","Q 2541 3569 2628 3553 \n","L 2631 2963 \n","z\n","\" transform=\"scale(0.015625)\"/>\n","       <path id=\"DejaVuSans-61\" d=\"M 2194 1759 \n","Q 1497 1759 1228 1600 \n","Q 959 1441 959 1056 \n","Q 959 750 1161 570 \n","Q 1363 391 1709 391 \n","Q 2188 391 2477 730 \n","Q 2766 1069 2766 1631 \n","L 2766 1759 \n","L 2194 1759 \n","z\n","M 3341 1997 \n","L 3341 0 \n","L 2766 0 \n","L 2766 531 \n","Q 2569 213 2275 61 \n","Q 1981 -91 1556 -91 \n","Q 1019 -91 701 211 \n","Q 384 513 384 1019 \n","Q 384 1609 779 1909 \n","Q 1175 2209 1959 2209 \n","L 2766 2209 \n","L 2766 2266 \n","Q 2766 2663 2505 2880 \n","Q 2244 3097 1772 3097 \n","Q 1472 3097 1187 3025 \n","Q 903 2953 641 2809 \n","L 641 3341 \n","Q 956 3463 1253 3523 \n","Q 1550 3584 1831 3584 \n","Q 2591 3584 2966 3190 \n","Q 3341 2797 3341 1997 \n","z\n","\" transform=\"scale(0.015625)\"/>\n","       <path id=\"DejaVuSans-69\" d=\"M 603 3500 \n","L 1178 3500 \n","L 1178 0 \n","L 603 0 \n","L 603 3500 \n","z\n","M 603 4863 \n","L 1178 4863 \n","L 1178 4134 \n","L 603 4134 \n","L 603 4863 \n","z\n","\" transform=\"scale(0.015625)\"/>\n","       <path id=\"DejaVuSans-6e\" d=\"M 3513 2113 \n","L 3513 0 \n","L 2938 0 \n","L 2938 2094 \n","Q 2938 2591 2744 2837 \n","Q 2550 3084 2163 3084 \n","Q 1697 3084 1428 2787 \n","Q 1159 2491 1159 1978 \n","L 1159 0 \n","L 581 0 \n","L 581 3500 \n","L 1159 3500 \n","L 1159 2956 \n","Q 1366 3272 1645 3428 \n","Q 1925 3584 2291 3584 \n","Q 2894 3584 3203 3211 \n","Q 3513 2838 3513 2113 \n","z\n","\" transform=\"scale(0.015625)\"/>\n","      </defs>\n","      <use xlink:href=\"#DejaVuSans-74\"/>\n","      <use xlink:href=\"#DejaVuSans-72\" x=\"39.208984\"/>\n","      <use xlink:href=\"#DejaVuSans-61\" x=\"80.322266\"/>\n","      <use xlink:href=\"#DejaVuSans-69\" x=\"141.601562\"/>\n","      <use xlink:href=\"#DejaVuSans-6e\" x=\"169.384766\"/>\n","     </g>\n","    </g>\n","    <g id=\"line2d_22\">\n","     <path d=\"M 60.378125 128.520312 \n","L 70.378125 128.520312 \n","L 80.378125 128.520312 \n","\" style=\"fill: none; stroke-dasharray: 5.55,2.4; stroke-dashoffset: 0; stroke: #bf00bf; stroke-width: 1.5\"/>\n","    </g>\n","    <g id=\"text_13\">\n","     <!-- test -->\n","     <g transform=\"translate(88.378125 132.020312)scale(0.1 -0.1)\">\n","      <use xlink:href=\"#DejaVuSans-74\"/>\n","      <use xlink:href=\"#DejaVuSans-65\" x=\"39.208984\"/>\n","      <use xlink:href=\"#DejaVuSans-73\" x=\"100.732422\"/>\n","      <use xlink:href=\"#DejaVuSans-74\" x=\"152.832031\"/>\n","     </g>\n","    </g>\n","   </g>\n","  </g>\n"," </g>\n"," <defs>\n","  <clipPath id=\"p5ed042e645\">\n","   <rect x=\"51.378125\" y=\"7.2\" width=\"195.3\" height=\"135.9\"/>\n","  </clipPath>\n"," </defs>\n","</svg>\n"],"text/plain":["<Figure size 252x180 with 1 Axes>"]},"metadata":{"needs_background":"light"},"output_type":"display_data"}],"source":["train_concise(0)"],"id":"oxh8fB35hB9E"},{"cell_type":"code","execution_count":null,"metadata":{"execution":{"iopub.execute_input":"2022-07-31T02:23:07.968650Z","iopub.status.busy":"2022-07-31T02:23:07.968162Z","iopub.status.idle":"2022-07-31T02:23:17.124533Z","shell.execute_reply":"2022-07-31T02:23:17.123862Z"},"origin_pos":31,"tab":["pytorch"],"outputId":"344001d5-8b4a-4ad7-98eb-ab0dd4d122d2","id":"oqZWJbBshB9E"},"outputs":[{"name":"stdout","output_type":"stream","text":["w的L2范数： 0.7186794877052307\n"]},{"data":{"image/svg+xml":["<?xml version=\"1.0\" encoding=\"utf-8\" standalone=\"no\"?>\n","<!DOCTYPE svg PUBLIC \"-//W3C//DTD SVG 1.1//EN\"\n","  \"http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd\">\n","<svg xmlns:xlink=\"http://www.w3.org/1999/xlink\" width=\"263.421875pt\" height=\"180.65625pt\" viewBox=\"0 0 263.421875 180.65625\" xmlns=\"http://www.w3.org/2000/svg\" version=\"1.1\">\n"," <metadata>\n","  <rdf:RDF xmlns:dc=\"http://purl.org/dc/elements/1.1/\" xmlns:cc=\"http://creativecommons.org/ns#\" xmlns:rdf=\"http://www.w3.org/1999/02/22-rdf-syntax-ns#\">\n","   <cc:Work>\n","    <dc:type rdf:resource=\"http://purl.org/dc/dcmitype/StillImage\"/>\n","    <dc:date>2022-07-31T02:23:17.007647</dc:date>\n","    <dc:format>image/svg+xml</dc:format>\n","    <dc:creator>\n","     <cc:Agent>\n","      <dc:title>Matplotlib v3.5.1, https://matplotlib.org/</dc:title>\n","     </cc:Agent>\n","    </dc:creator>\n","   </cc:Work>\n","  </rdf:RDF>\n"," </metadata>\n"," <defs>\n","  <style type=\"text/css\">*{stroke-linejoin: round; stroke-linecap: butt}</style>\n"," </defs>\n"," <g id=\"figure_1\">\n","  <g id=\"patch_1\">\n","   <path d=\"M 0 180.65625 \n","L 263.421875 180.65625 \n","L 263.421875 0 \n","L 0 0 \n","L 0 180.65625 \n","z\n","\" style=\"fill: none\"/>\n","  </g>\n","  <g id=\"axes_1\">\n","   <g id=\"patch_2\">\n","    <path d=\"M 51.378125 143.1 \n","L 246.678125 143.1 \n","L 246.678125 7.2 \n","L 51.378125 7.2 \n","z\n","\" style=\"fill: #ffffff\"/>\n","   </g>\n","   <g id=\"matplotlib.axis_1\">\n","    <g id=\"xtick_1\">\n","     <g id=\"line2d_1\">\n","      <path d=\"M 82.214967 143.1 \n","L 82.214967 7.2 \n","\" clip-path=\"url(#p97ac790d64)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n","     </g>\n","     <g id=\"line2d_2\">\n","      <defs>\n","       <path id=\"m4856621ce4\" d=\"M 0 0 \n","L 0 3.5 \n","\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n","      </defs>\n","      <g>\n","       <use xlink:href=\"#m4856621ce4\" x=\"82.214967\" y=\"143.1\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n","      </g>\n","     </g>\n","     <g id=\"text_1\">\n","      <!-- 20 -->\n","      <g transform=\"translate(75.852467 157.698438)scale(0.1 -0.1)\">\n","       <defs>\n","        <path id=\"DejaVuSans-32\" d=\"M 1228 531 \n","L 3431 531 \n","L 3431 0 \n","L 469 0 \n","L 469 531 \n","Q 828 903 1448 1529 \n","Q 2069 2156 2228 2338 \n","Q 2531 2678 2651 2914 \n","Q 2772 3150 2772 3378 \n","Q 2772 3750 2511 3984 \n","Q 2250 4219 1831 4219 \n","Q 1534 4219 1204 4116 \n","Q 875 4013 500 3803 \n","L 500 4441 \n","Q 881 4594 1212 4672 \n","Q 1544 4750 1819 4750 \n","Q 2544 4750 2975 4387 \n","Q 3406 4025 3406 3419 \n","Q 3406 3131 3298 2873 \n","Q 3191 2616 2906 2266 \n","Q 2828 2175 2409 1742 \n","Q 1991 1309 1228 531 \n","z\n","\" transform=\"scale(0.015625)\"/>\n","        <path id=\"DejaVuSans-30\" d=\"M 2034 4250 \n","Q 1547 4250 1301 3770 \n","Q 1056 3291 1056 2328 \n","Q 1056 1369 1301 889 \n","Q 1547 409 2034 409 \n","Q 2525 409 2770 889 \n","Q 3016 1369 3016 2328 \n","Q 3016 3291 2770 3770 \n","Q 2525 4250 2034 4250 \n","z\n","M 2034 4750 \n","Q 2819 4750 3233 4129 \n","Q 3647 3509 3647 2328 \n","Q 3647 1150 3233 529 \n","Q 2819 -91 2034 -91 \n","Q 1250 -91 836 529 \n","Q 422 1150 422 2328 \n","Q 422 3509 836 4129 \n","Q 1250 4750 2034 4750 \n","z\n","\" transform=\"scale(0.015625)\"/>\n","       </defs>\n","       <use xlink:href=\"#DejaVuSans-32\"/>\n","       <use xlink:href=\"#DejaVuSans-30\" x=\"63.623047\"/>\n","      </g>\n","     </g>\n","    </g>\n","    <g id=\"xtick_2\">\n","     <g id=\"line2d_3\">\n","      <path d=\"M 123.330757 143.1 \n","L 123.330757 7.2 \n","\" clip-path=\"url(#p97ac790d64)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n","     </g>\n","     <g id=\"line2d_4\">\n","      <g>\n","       <use xlink:href=\"#m4856621ce4\" x=\"123.330757\" y=\"143.1\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n","      </g>\n","     </g>\n","     <g id=\"text_2\">\n","      <!-- 40 -->\n","      <g transform=\"translate(116.968257 157.698438)scale(0.1 -0.1)\">\n","       <defs>\n","        <path id=\"DejaVuSans-34\" d=\"M 2419 4116 \n","L 825 1625 \n","L 2419 1625 \n","L 2419 4116 \n","z\n","M 2253 4666 \n","L 3047 4666 \n","L 3047 1625 \n","L 3713 1625 \n","L 3713 1100 \n","L 3047 1100 \n","L 3047 0 \n","L 2419 0 \n","L 2419 1100 \n","L 313 1100 \n","L 313 1709 \n","L 2253 4666 \n","z\n","\" transform=\"scale(0.015625)\"/>\n","       </defs>\n","       <use xlink:href=\"#DejaVuSans-34\"/>\n","       <use xlink:href=\"#DejaVuSans-30\" x=\"63.623047\"/>\n","      </g>\n","     </g>\n","    </g>\n","    <g id=\"xtick_3\">\n","     <g id=\"line2d_5\">\n","      <path d=\"M 164.446546 143.1 \n","L 164.446546 7.2 \n","\" clip-path=\"url(#p97ac790d64)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n","     </g>\n","     <g id=\"line2d_6\">\n","      <g>\n","       <use xlink:href=\"#m4856621ce4\" x=\"164.446546\" y=\"143.1\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n","      </g>\n","     </g>\n","     <g id=\"text_3\">\n","      <!-- 60 -->\n","      <g transform=\"translate(158.084046 157.698438)scale(0.1 -0.1)\">\n","       <defs>\n","        <path id=\"DejaVuSans-36\" d=\"M 2113 2584 \n","Q 1688 2584 1439 2293 \n","Q 1191 2003 1191 1497 \n","Q 1191 994 1439 701 \n","Q 1688 409 2113 409 \n","Q 2538 409 2786 701 \n","Q 3034 994 3034 1497 \n","Q 3034 2003 2786 2293 \n","Q 2538 2584 2113 2584 \n","z\n","M 3366 4563 \n","L 3366 3988 \n","Q 3128 4100 2886 4159 \n","Q 2644 4219 2406 4219 \n","Q 1781 4219 1451 3797 \n","Q 1122 3375 1075 2522 \n","Q 1259 2794 1537 2939 \n","Q 1816 3084 2150 3084 \n","Q 2853 3084 3261 2657 \n","Q 3669 2231 3669 1497 \n","Q 3669 778 3244 343 \n","Q 2819 -91 2113 -91 \n","Q 1303 -91 875 529 \n","Q 447 1150 447 2328 \n","Q 447 3434 972 4092 \n","Q 1497 4750 2381 4750 \n","Q 2619 4750 2861 4703 \n","Q 3103 4656 3366 4563 \n","z\n","\" transform=\"scale(0.015625)\"/>\n","       </defs>\n","       <use xlink:href=\"#DejaVuSans-36\"/>\n","       <use xlink:href=\"#DejaVuSans-30\" x=\"63.623047\"/>\n","      </g>\n","     </g>\n","    </g>\n","    <g id=\"xtick_4\">\n","     <g id=\"line2d_7\">\n","      <path d=\"M 205.562336 143.1 \n","L 205.562336 7.2 \n","\" clip-path=\"url(#p97ac790d64)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n","     </g>\n","     <g id=\"line2d_8\">\n","      <g>\n","       <use xlink:href=\"#m4856621ce4\" x=\"205.562336\" y=\"143.1\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n","      </g>\n","     </g>\n","     <g id=\"text_4\">\n","      <!-- 80 -->\n","      <g transform=\"translate(199.199836 157.698438)scale(0.1 -0.1)\">\n","       <defs>\n","        <path id=\"DejaVuSans-38\" d=\"M 2034 2216 \n","Q 1584 2216 1326 1975 \n","Q 1069 1734 1069 1313 \n","Q 1069 891 1326 650 \n","Q 1584 409 2034 409 \n","Q 2484 409 2743 651 \n","Q 3003 894 3003 1313 \n","Q 3003 1734 2745 1975 \n","Q 2488 2216 2034 2216 \n","z\n","M 1403 2484 \n","Q 997 2584 770 2862 \n","Q 544 3141 544 3541 \n","Q 544 4100 942 4425 \n","Q 1341 4750 2034 4750 \n","Q 2731 4750 3128 4425 \n","Q 3525 4100 3525 3541 \n","Q 3525 3141 3298 2862 \n","Q 3072 2584 2669 2484 \n","Q 3125 2378 3379 2068 \n","Q 3634 1759 3634 1313 \n","Q 3634 634 3220 271 \n","Q 2806 -91 2034 -91 \n","Q 1263 -91 848 271 \n","Q 434 634 434 1313 \n","Q 434 1759 690 2068 \n","Q 947 2378 1403 2484 \n","z\n","M 1172 3481 \n","Q 1172 3119 1398 2916 \n","Q 1625 2713 2034 2713 \n","Q 2441 2713 2670 2916 \n","Q 2900 3119 2900 3481 \n","Q 2900 3844 2670 4047 \n","Q 2441 4250 2034 4250 \n","Q 1625 4250 1398 4047 \n","Q 1172 3844 1172 3481 \n","z\n","\" transform=\"scale(0.015625)\"/>\n","       </defs>\n","       <use xlink:href=\"#DejaVuSans-38\"/>\n","       <use xlink:href=\"#DejaVuSans-30\" x=\"63.623047\"/>\n","      </g>\n","     </g>\n","    </g>\n","    <g id=\"xtick_5\">\n","     <g id=\"line2d_9\">\n","      <path d=\"M 246.678125 143.1 \n","L 246.678125 7.2 \n","\" clip-path=\"url(#p97ac790d64)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n","     </g>\n","     <g id=\"line2d_10\">\n","      <g>\n","       <use xlink:href=\"#m4856621ce4\" x=\"246.678125\" y=\"143.1\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n","      </g>\n","     </g>\n","     <g id=\"text_5\">\n","      <!-- 100 -->\n","      <g transform=\"translate(237.134375 157.698438)scale(0.1 -0.1)\">\n","       <defs>\n","        <path id=\"DejaVuSans-31\" d=\"M 794 531 \n","L 1825 531 \n","L 1825 4091 \n","L 703 3866 \n","L 703 4441 \n","L 1819 4666 \n","L 2450 4666 \n","L 2450 531 \n","L 3481 531 \n","L 3481 0 \n","L 794 0 \n","L 794 531 \n","z\n","\" transform=\"scale(0.015625)\"/>\n","       </defs>\n","       <use xlink:href=\"#DejaVuSans-31\"/>\n","       <use xlink:href=\"#DejaVuSans-30\" x=\"63.623047\"/>\n","       <use xlink:href=\"#DejaVuSans-30\" x=\"127.246094\"/>\n","      </g>\n","     </g>\n","    </g>\n","    <g id=\"text_6\">\n","     <!-- epochs -->\n","     <g transform=\"translate(131.195312 171.376563)scale(0.1 -0.1)\">\n","      <defs>\n","       <path id=\"DejaVuSans-65\" d=\"M 3597 1894 \n","L 3597 1613 \n","L 953 1613 \n","Q 991 1019 1311 708 \n","Q 1631 397 2203 397 \n","Q 2534 397 2845 478 \n","Q 3156 559 3463 722 \n","L 3463 178 \n","Q 3153 47 2828 -22 \n","Q 2503 -91 2169 -91 \n","Q 1331 -91 842 396 \n","Q 353 884 353 1716 \n","Q 353 2575 817 3079 \n","Q 1281 3584 2069 3584 \n","Q 2775 3584 3186 3129 \n","Q 3597 2675 3597 1894 \n","z\n","M 3022 2063 \n","Q 3016 2534 2758 2815 \n","Q 2500 3097 2075 3097 \n","Q 1594 3097 1305 2825 \n","Q 1016 2553 972 2059 \n","L 3022 2063 \n","z\n","\" transform=\"scale(0.015625)\"/>\n","       <path id=\"DejaVuSans-70\" d=\"M 1159 525 \n","L 1159 -1331 \n","L 581 -1331 \n","L 581 3500 \n","L 1159 3500 \n","L 1159 2969 \n","Q 1341 3281 1617 3432 \n","Q 1894 3584 2278 3584 \n","Q 2916 3584 3314 3078 \n","Q 3713 2572 3713 1747 \n","Q 3713 922 3314 415 \n","Q 2916 -91 2278 -91 \n","Q 1894 -91 1617 61 \n","Q 1341 213 1159 525 \n","z\n","M 3116 1747 \n","Q 3116 2381 2855 2742 \n","Q 2594 3103 2138 3103 \n","Q 1681 3103 1420 2742 \n","Q 1159 2381 1159 1747 \n","Q 1159 1113 1420 752 \n","Q 1681 391 2138 391 \n","Q 2594 391 2855 752 \n","Q 3116 1113 3116 1747 \n","z\n","\" transform=\"scale(0.015625)\"/>\n","       <path id=\"DejaVuSans-6f\" d=\"M 1959 3097 \n","Q 1497 3097 1228 2736 \n","Q 959 2375 959 1747 \n","Q 959 1119 1226 758 \n","Q 1494 397 1959 397 \n","Q 2419 397 2687 759 \n","Q 2956 1122 2956 1747 \n","Q 2956 2369 2687 2733 \n","Q 2419 3097 1959 3097 \n","z\n","M 1959 3584 \n","Q 2709 3584 3137 3096 \n","Q 3566 2609 3566 1747 \n","Q 3566 888 3137 398 \n","Q 2709 -91 1959 -91 \n","Q 1206 -91 779 398 \n","Q 353 888 353 1747 \n","Q 353 2609 779 3096 \n","Q 1206 3584 1959 3584 \n","z\n","\" transform=\"scale(0.015625)\"/>\n","       <path id=\"DejaVuSans-63\" d=\"M 3122 3366 \n","L 3122 2828 \n","Q 2878 2963 2633 3030 \n","Q 2388 3097 2138 3097 \n","Q 1578 3097 1268 2742 \n","Q 959 2388 959 1747 \n","Q 959 1106 1268 751 \n","Q 1578 397 2138 397 \n","Q 2388 397 2633 464 \n","Q 2878 531 3122 666 \n","L 3122 134 \n","Q 2881 22 2623 -34 \n","Q 2366 -91 2075 -91 \n","Q 1284 -91 818 406 \n","Q 353 903 353 1747 \n","Q 353 2603 823 3093 \n","Q 1294 3584 2113 3584 \n","Q 2378 3584 2631 3529 \n","Q 2884 3475 3122 3366 \n","z\n","\" transform=\"scale(0.015625)\"/>\n","       <path id=\"DejaVuSans-68\" d=\"M 3513 2113 \n","L 3513 0 \n","L 2938 0 \n","L 2938 2094 \n","Q 2938 2591 2744 2837 \n","Q 2550 3084 2163 3084 \n","Q 1697 3084 1428 2787 \n","Q 1159 2491 1159 1978 \n","L 1159 0 \n","L 581 0 \n","L 581 4863 \n","L 1159 4863 \n","L 1159 2956 \n","Q 1366 3272 1645 3428 \n","Q 1925 3584 2291 3584 \n","Q 2894 3584 3203 3211 \n","Q 3513 2838 3513 2113 \n","z\n","\" transform=\"scale(0.015625)\"/>\n","       <path id=\"DejaVuSans-73\" d=\"M 2834 3397 \n","L 2834 2853 \n","Q 2591 2978 2328 3040 \n","Q 2066 3103 1784 3103 \n","Q 1356 3103 1142 2972 \n","Q 928 2841 928 2578 \n","Q 928 2378 1081 2264 \n","Q 1234 2150 1697 2047 \n","L 1894 2003 \n","Q 2506 1872 2764 1633 \n","Q 3022 1394 3022 966 \n","Q 3022 478 2636 193 \n","Q 2250 -91 1575 -91 \n","Q 1294 -91 989 -36 \n","Q 684 19 347 128 \n","L 347 722 \n","Q 666 556 975 473 \n","Q 1284 391 1588 391 \n","Q 1994 391 2212 530 \n","Q 2431 669 2431 922 \n","Q 2431 1156 2273 1281 \n","Q 2116 1406 1581 1522 \n","L 1381 1569 \n","Q 847 1681 609 1914 \n","Q 372 2147 372 2553 \n","Q 372 3047 722 3315 \n","Q 1072 3584 1716 3584 \n","Q 2034 3584 2315 3537 \n","Q 2597 3491 2834 3397 \n","z\n","\" transform=\"scale(0.015625)\"/>\n","      </defs>\n","      <use xlink:href=\"#DejaVuSans-65\"/>\n","      <use xlink:href=\"#DejaVuSans-70\" x=\"61.523438\"/>\n","      <use xlink:href=\"#DejaVuSans-6f\" x=\"125\"/>\n","      <use xlink:href=\"#DejaVuSans-63\" x=\"186.181641\"/>\n","      <use xlink:href=\"#DejaVuSans-68\" x=\"241.162109\"/>\n","      <use xlink:href=\"#DejaVuSans-73\" x=\"304.541016\"/>\n","     </g>\n","    </g>\n","   </g>\n","   <g id=\"matplotlib.axis_2\">\n","    <g id=\"ytick_1\">\n","     <g id=\"line2d_11\">\n","      <path d=\"M 51.378125 131.054124 \n","L 246.678125 131.054124 \n","\" clip-path=\"url(#p97ac790d64)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n","     </g>\n","     <g id=\"line2d_12\">\n","      <defs>\n","       <path id=\"m5d32c10c90\" d=\"M 0 0 \n","L -3.5 0 \n","\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n","      </defs>\n","      <g>\n","       <use xlink:href=\"#m5d32c10c90\" x=\"51.378125\" y=\"131.054124\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n","      </g>\n","     </g>\n","     <g id=\"text_7\">\n","      <!-- $\\mathdefault{10^{-1}}$ -->\n","      <g transform=\"translate(20.878125 134.853343)scale(0.1 -0.1)\">\n","       <defs>\n","        <path id=\"DejaVuSans-2212\" d=\"M 678 2272 \n","L 4684 2272 \n","L 4684 1741 \n","L 678 1741 \n","L 678 2272 \n","z\n","\" transform=\"scale(0.015625)\"/>\n","       </defs>\n","       <use xlink:href=\"#DejaVuSans-31\" transform=\"translate(0 0.684375)\"/>\n","       <use xlink:href=\"#DejaVuSans-30\" transform=\"translate(63.623047 0.684375)\"/>\n","       <use xlink:href=\"#DejaVuSans-2212\" transform=\"translate(128.203125 38.965625)scale(0.7)\"/>\n","       <use xlink:href=\"#DejaVuSans-31\" transform=\"translate(186.855469 38.965625)scale(0.7)\"/>\n","      </g>\n","     </g>\n","    </g>\n","    <g id=\"ytick_2\">\n","     <g id=\"line2d_13\">\n","      <path d=\"M 51.378125 95.094219 \n","L 246.678125 95.094219 \n","\" clip-path=\"url(#p97ac790d64)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n","     </g>\n","     <g id=\"line2d_14\">\n","      <g>\n","       <use xlink:href=\"#m5d32c10c90\" x=\"51.378125\" y=\"95.094219\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n","      </g>\n","     </g>\n","     <g id=\"text_8\">\n","      <!-- $\\mathdefault{10^{0}}$ -->\n","      <g transform=\"translate(26.778125 98.893437)scale(0.1 -0.1)\">\n","       <use xlink:href=\"#DejaVuSans-31\" transform=\"translate(0 0.765625)\"/>\n","       <use xlink:href=\"#DejaVuSans-30\" transform=\"translate(63.623047 0.765625)\"/>\n","       <use xlink:href=\"#DejaVuSans-30\" transform=\"translate(128.203125 39.046875)scale(0.7)\"/>\n","      </g>\n","     </g>\n","    </g>\n","    <g id=\"ytick_3\">\n","     <g id=\"line2d_15\">\n","      <path d=\"M 51.378125 59.134313 \n","L 246.678125 59.134313 \n","\" clip-path=\"url(#p97ac790d64)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n","     </g>\n","     <g id=\"line2d_16\">\n","      <g>\n","       <use xlink:href=\"#m5d32c10c90\" x=\"51.378125\" y=\"59.134313\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n","      </g>\n","     </g>\n","     <g id=\"text_9\">\n","      <!-- $\\mathdefault{10^{1}}$ -->\n","      <g transform=\"translate(26.778125 62.933532)scale(0.1 -0.1)\">\n","       <use xlink:href=\"#DejaVuSans-31\" transform=\"translate(0 0.684375)\"/>\n","       <use xlink:href=\"#DejaVuSans-30\" transform=\"translate(63.623047 0.684375)\"/>\n","       <use xlink:href=\"#DejaVuSans-31\" transform=\"translate(128.203125 38.965625)scale(0.7)\"/>\n","      </g>\n","     </g>\n","    </g>\n","    <g id=\"ytick_4\">\n","     <g id=\"line2d_17\">\n","      <path d=\"M 51.378125 23.174408 \n","L 246.678125 23.174408 \n","\" clip-path=\"url(#p97ac790d64)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n","     </g>\n","     <g id=\"line2d_18\">\n","      <g>\n","       <use xlink:href=\"#m5d32c10c90\" x=\"51.378125\" y=\"23.174408\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n","      </g>\n","     </g>\n","     <g id=\"text_10\">\n","      <!-- $\\mathdefault{10^{2}}$ -->\n","      <g transform=\"translate(26.778125 26.973626)scale(0.1 -0.1)\">\n","       <use xlink:href=\"#DejaVuSans-31\" transform=\"translate(0 0.765625)\"/>\n","       <use xlink:href=\"#DejaVuSans-30\" transform=\"translate(63.623047 0.765625)\"/>\n","       <use xlink:href=\"#DejaVuSans-32\" transform=\"translate(128.203125 39.046875)scale(0.7)\"/>\n","      </g>\n","     </g>\n","    </g>\n","    <g id=\"ytick_5\">\n","     <g id=\"line2d_19\">\n","      <defs>\n","       <path id=\"mfa9bd9e3c7\" d=\"M 0 0 \n","L -2 0 \n","\" style=\"stroke: #000000; stroke-width: 0.6\"/>\n","      </defs>\n","      <g>\n","       <use xlink:href=\"#mfa9bd9e3c7\" x=\"51.378125\" y=\"141.879135\" style=\"stroke: #000000; stroke-width: 0.6\"/>\n","      </g>\n","     </g>\n","    </g>\n","    <g id=\"ytick_6\">\n","     <g id=\"line2d_20\">\n","      <g>\n","       <use xlink:href=\"#mfa9bd9e3c7\" x=\"51.378125\" y=\"139.031784\" style=\"stroke: #000000; stroke-width: 0.6\"/>\n","      </g>\n","     </g>\n","    </g>\n","    <g id=\"ytick_7\">\n","     <g id=\"line2d_21\">\n","      <g>\n","       <use xlink:href=\"#mfa9bd9e3c7\" x=\"51.378125\" y=\"136.624384\" style=\"stroke: #000000; stroke-width: 0.6\"/>\n","      </g>\n","     </g>\n","    </g>\n","    <g id=\"ytick_8\">\n","     <g id=\"line2d_22\">\n","      <g>\n","       <use xlink:href=\"#mfa9bd9e3c7\" x=\"51.378125\" y=\"134.538999\" style=\"stroke: #000000; stroke-width: 0.6\"/>\n","      </g>\n","     </g>\n","    </g>\n","    <g id=\"ytick_9\">\n","     <g id=\"line2d_23\">\n","      <g>\n","       <use xlink:href=\"#mfa9bd9e3c7\" x=\"51.378125\" y=\"132.699559\" style=\"stroke: #000000; stroke-width: 0.6\"/>\n","      </g>\n","     </g>\n","    </g>\n","    <g id=\"ytick_10\">\n","     <g id=\"line2d_24\">\n","      <g>\n","       <use xlink:href=\"#mfa9bd9e3c7\" x=\"51.378125\" y=\"120.229114\" style=\"stroke: #000000; stroke-width: 0.6\"/>\n","      </g>\n","     </g>\n","    </g>\n","    <g id=\"ytick_11\">\n","     <g id=\"line2d_25\">\n","      <g>\n","       <use xlink:href=\"#mfa9bd9e3c7\" x=\"51.378125\" y=\"113.896889\" style=\"stroke: #000000; stroke-width: 0.6\"/>\n","      </g>\n","     </g>\n","    </g>\n","    <g id=\"ytick_12\">\n","     <g id=\"line2d_26\">\n","      <g>\n","       <use xlink:href=\"#mfa9bd9e3c7\" x=\"51.378125\" y=\"109.404104\" style=\"stroke: #000000; stroke-width: 0.6\"/>\n","      </g>\n","     </g>\n","    </g>\n","    <g id=\"ytick_13\">\n","     <g id=\"line2d_27\">\n","      <g>\n","       <use xlink:href=\"#mfa9bd9e3c7\" x=\"51.378125\" y=\"105.919229\" style=\"stroke: #000000; stroke-width: 0.6\"/>\n","      </g>\n","     </g>\n","    </g>\n","    <g id=\"ytick_14\">\n","     <g id=\"line2d_28\">\n","      <g>\n","       <use xlink:href=\"#mfa9bd9e3c7\" x=\"51.378125\" y=\"103.071879\" style=\"stroke: #000000; stroke-width: 0.6\"/>\n","      </g>\n","     </g>\n","    </g>\n","    <g id=\"ytick_15\">\n","     <g id=\"line2d_29\">\n","      <g>\n","       <use xlink:href=\"#mfa9bd9e3c7\" x=\"51.378125\" y=\"100.664479\" style=\"stroke: #000000; stroke-width: 0.6\"/>\n","      </g>\n","     </g>\n","    </g>\n","    <g id=\"ytick_16\">\n","     <g id=\"line2d_30\">\n","      <g>\n","       <use xlink:href=\"#mfa9bd9e3c7\" x=\"51.378125\" y=\"98.579094\" style=\"stroke: #000000; stroke-width: 0.6\"/>\n","      </g>\n","     </g>\n","    </g>\n","    <g id=\"ytick_17\">\n","     <g id=\"line2d_31\">\n","      <g>\n","       <use xlink:href=\"#mfa9bd9e3c7\" x=\"51.378125\" y=\"96.739654\" style=\"stroke: #000000; stroke-width: 0.6\"/>\n","      </g>\n","     </g>\n","    </g>\n","    <g id=\"ytick_18\">\n","     <g id=\"line2d_32\">\n","      <g>\n","       <use xlink:href=\"#mfa9bd9e3c7\" x=\"51.378125\" y=\"84.269209\" style=\"stroke: #000000; stroke-width: 0.6\"/>\n","      </g>\n","     </g>\n","    </g>\n","    <g id=\"ytick_19\">\n","     <g id=\"line2d_33\">\n","      <g>\n","       <use xlink:href=\"#mfa9bd9e3c7\" x=\"51.378125\" y=\"77.936983\" style=\"stroke: #000000; stroke-width: 0.6\"/>\n","      </g>\n","     </g>\n","    </g>\n","    <g id=\"ytick_20\">\n","     <g id=\"line2d_34\">\n","      <g>\n","       <use xlink:href=\"#mfa9bd9e3c7\" x=\"51.378125\" y=\"73.444198\" style=\"stroke: #000000; stroke-width: 0.6\"/>\n","      </g>\n","     </g>\n","    </g>\n","    <g id=\"ytick_21\">\n","     <g id=\"line2d_35\">\n","      <g>\n","       <use xlink:href=\"#mfa9bd9e3c7\" x=\"51.378125\" y=\"69.959323\" style=\"stroke: #000000; stroke-width: 0.6\"/>\n","      </g>\n","     </g>\n","    </g>\n","    <g id=\"ytick_22\">\n","     <g id=\"line2d_36\">\n","      <g>\n","       <use xlink:href=\"#mfa9bd9e3c7\" x=\"51.378125\" y=\"67.111973\" style=\"stroke: #000000; stroke-width: 0.6\"/>\n","      </g>\n","     </g>\n","    </g>\n","    <g id=\"ytick_23\">\n","     <g id=\"line2d_37\">\n","      <g>\n","       <use xlink:href=\"#mfa9bd9e3c7\" x=\"51.378125\" y=\"64.704573\" style=\"stroke: #000000; stroke-width: 0.6\"/>\n","      </g>\n","     </g>\n","    </g>\n","    <g id=\"ytick_24\">\n","     <g id=\"line2d_38\">\n","      <g>\n","       <use xlink:href=\"#mfa9bd9e3c7\" x=\"51.378125\" y=\"62.619188\" style=\"stroke: #000000; stroke-width: 0.6\"/>\n","      </g>\n","     </g>\n","    </g>\n","    <g id=\"ytick_25\">\n","     <g id=\"line2d_39\">\n","      <g>\n","       <use xlink:href=\"#mfa9bd9e3c7\" x=\"51.378125\" y=\"60.779748\" style=\"stroke: #000000; stroke-width: 0.6\"/>\n","      </g>\n","     </g>\n","    </g>\n","    <g id=\"ytick_26\">\n","     <g id=\"line2d_40\">\n","      <g>\n","       <use xlink:href=\"#mfa9bd9e3c7\" x=\"51.378125\" y=\"48.309303\" style=\"stroke: #000000; stroke-width: 0.6\"/>\n","      </g>\n","     </g>\n","    </g>\n","    <g id=\"ytick_27\">\n","     <g id=\"line2d_41\">\n","      <g>\n","       <use xlink:href=\"#mfa9bd9e3c7\" x=\"51.378125\" y=\"41.977078\" style=\"stroke: #000000; stroke-width: 0.6\"/>\n","      </g>\n","     </g>\n","    </g>\n","    <g id=\"ytick_28\">\n","     <g id=\"line2d_42\">\n","      <g>\n","       <use xlink:href=\"#mfa9bd9e3c7\" x=\"51.378125\" y=\"37.484293\" style=\"stroke: #000000; stroke-width: 0.6\"/>\n","      </g>\n","     </g>\n","    </g>\n","    <g id=\"ytick_29\">\n","     <g id=\"line2d_43\">\n","      <g>\n","       <use xlink:href=\"#mfa9bd9e3c7\" x=\"51.378125\" y=\"33.999418\" style=\"stroke: #000000; stroke-width: 0.6\"/>\n","      </g>\n","     </g>\n","    </g>\n","    <g id=\"ytick_30\">\n","     <g id=\"line2d_44\">\n","      <g>\n","       <use xlink:href=\"#mfa9bd9e3c7\" x=\"51.378125\" y=\"31.152068\" style=\"stroke: #000000; stroke-width: 0.6\"/>\n","      </g>\n","     </g>\n","    </g>\n","    <g id=\"ytick_31\">\n","     <g id=\"line2d_45\">\n","      <g>\n","       <use xlink:href=\"#mfa9bd9e3c7\" x=\"51.378125\" y=\"28.744667\" style=\"stroke: #000000; stroke-width: 0.6\"/>\n","      </g>\n","     </g>\n","    </g>\n","    <g id=\"ytick_32\">\n","     <g id=\"line2d_46\">\n","      <g>\n","       <use xlink:href=\"#mfa9bd9e3c7\" x=\"51.378125\" y=\"26.659282\" style=\"stroke: #000000; stroke-width: 0.6\"/>\n","      </g>\n","     </g>\n","    </g>\n","    <g id=\"ytick_33\">\n","     <g id=\"line2d_47\">\n","      <g>\n","       <use xlink:href=\"#mfa9bd9e3c7\" x=\"51.378125\" y=\"24.819843\" style=\"stroke: #000000; stroke-width: 0.6\"/>\n","      </g>\n","     </g>\n","    </g>\n","    <g id=\"ytick_34\">\n","     <g id=\"line2d_48\">\n","      <g>\n","       <use xlink:href=\"#mfa9bd9e3c7\" x=\"51.378125\" y=\"12.349397\" style=\"stroke: #000000; stroke-width: 0.6\"/>\n","      </g>\n","     </g>\n","    </g>\n","    <g id=\"text_11\">\n","     <!-- loss -->\n","     <g transform=\"translate(14.798437 84.807812)rotate(-90)scale(0.1 -0.1)\">\n","      <defs>\n","       <path id=\"DejaVuSans-6c\" d=\"M 603 4863 \n","L 1178 4863 \n","L 1178 0 \n","L 603 0 \n","L 603 4863 \n","z\n","\" transform=\"scale(0.015625)\"/>\n","      </defs>\n","      <use xlink:href=\"#DejaVuSans-6c\"/>\n","      <use xlink:href=\"#DejaVuSans-6f\" x=\"27.783203\"/>\n","      <use xlink:href=\"#DejaVuSans-73\" x=\"88.964844\"/>\n","      <use xlink:href=\"#DejaVuSans-73\" x=\"141.064453\"/>\n","     </g>\n","    </g>\n","   </g>\n","   <g id=\"line2d_49\">\n","    <path d=\"M 51.378125 66.609229 \n","L 61.657072 102.92158 \n","L 71.93602 124.166574 \n","L 82.214967 129.389785 \n","L 92.493914 130.171889 \n","L 102.772862 130.403184 \n","L 113.051809 130.637817 \n","L 123.330757 131.091839 \n","L 133.609704 131.591448 \n","L 143.888651 131.989853 \n","L 154.167599 132.540155 \n","L 164.446546 133.046585 \n","L 174.725493 133.574638 \n","L 185.004441 134.073878 \n","L 195.283388 134.510888 \n","L 205.562336 135.01178 \n","L 215.841283 135.521189 \n","L 226.12023 135.992137 \n","L 236.399178 136.381117 \n","L 246.678125 136.922727 \n","\" clip-path=\"url(#p97ac790d64)\" style=\"fill: none; stroke: #1f77b4; stroke-width: 1.5; stroke-linecap: square\"/>\n","   </g>\n","   <g id=\"line2d_50\">\n","    <path d=\"M 51.378125 13.377273 \n","L 61.657072 19.102244 \n","L 71.93602 24.613319 \n","L 82.214967 29.953459 \n","L 92.493914 35.111297 \n","L 102.772862 40.045542 \n","L 113.051809 44.698434 \n","L 123.330757 49.00995 \n","L 133.609704 52.909019 \n","L 143.888651 56.354886 \n","L 154.167599 59.31546 \n","L 164.446546 61.804721 \n","L 174.725493 63.873401 \n","L 185.004441 65.541152 \n","L 195.283388 66.885531 \n","L 205.562336 67.993916 \n","L 215.841283 68.902959 \n","L 226.12023 69.678381 \n","L 236.399178 70.352107 \n","L 246.678125 70.951613 \n","\" clip-path=\"url(#p97ac790d64)\" style=\"fill: none; stroke-dasharray: 5.55,2.4; stroke-dashoffset: 0; stroke: #bf00bf; stroke-width: 1.5\"/>\n","   </g>\n","   <g id=\"patch_3\">\n","    <path d=\"M 51.378125 143.1 \n","L 51.378125 7.2 \n","\" style=\"fill: none; stroke: #000000; stroke-width: 0.8; stroke-linejoin: miter; stroke-linecap: square\"/>\n","   </g>\n","   <g id=\"patch_4\">\n","    <path d=\"M 246.678125 143.1 \n","L 246.678125 7.2 \n","\" style=\"fill: none; stroke: #000000; stroke-width: 0.8; stroke-linejoin: miter; stroke-linecap: square\"/>\n","   </g>\n","   <g id=\"patch_5\">\n","    <path d=\"M 51.378125 143.1 \n","L 246.678125 143.1 \n","\" style=\"fill: none; stroke: #000000; stroke-width: 0.8; stroke-linejoin: miter; stroke-linecap: square\"/>\n","   </g>\n","   <g id=\"patch_6\">\n","    <path d=\"M 51.378125 7.2 \n","L 246.678125 7.2 \n","\" style=\"fill: none; stroke: #000000; stroke-width: 0.8; stroke-linejoin: miter; stroke-linecap: square\"/>\n","   </g>\n","   <g id=\"legend_1\">\n","    <g id=\"patch_7\">\n","     <path d=\"M 184.403125 44.55625 \n","L 239.678125 44.55625 \n","Q 241.678125 44.55625 241.678125 42.55625 \n","L 241.678125 14.2 \n","Q 241.678125 12.2 239.678125 12.2 \n","L 184.403125 12.2 \n","Q 182.403125 12.2 182.403125 14.2 \n","L 182.403125 42.55625 \n","Q 182.403125 44.55625 184.403125 44.55625 \n","z\n","\" style=\"fill: #ffffff; opacity: 0.8; stroke: #cccccc; stroke-linejoin: miter\"/>\n","    </g>\n","    <g id=\"line2d_51\">\n","     <path d=\"M 186.403125 20.298437 \n","L 196.403125 20.298437 \n","L 206.403125 20.298437 \n","\" style=\"fill: none; stroke: #1f77b4; stroke-width: 1.5; stroke-linecap: square\"/>\n","    </g>\n","    <g id=\"text_12\">\n","     <!-- train -->\n","     <g transform=\"translate(214.403125 23.798437)scale(0.1 -0.1)\">\n","      <defs>\n","       <path id=\"DejaVuSans-74\" d=\"M 1172 4494 \n","L 1172 3500 \n","L 2356 3500 \n","L 2356 3053 \n","L 1172 3053 \n","L 1172 1153 \n","Q 1172 725 1289 603 \n","Q 1406 481 1766 481 \n","L 2356 481 \n","L 2356 0 \n","L 1766 0 \n","Q 1100 0 847 248 \n","Q 594 497 594 1153 \n","L 594 3053 \n","L 172 3053 \n","L 172 3500 \n","L 594 3500 \n","L 594 4494 \n","L 1172 4494 \n","z\n","\" transform=\"scale(0.015625)\"/>\n","       <path id=\"DejaVuSans-72\" d=\"M 2631 2963 \n","Q 2534 3019 2420 3045 \n","Q 2306 3072 2169 3072 \n","Q 1681 3072 1420 2755 \n","Q 1159 2438 1159 1844 \n","L 1159 0 \n","L 581 0 \n","L 581 3500 \n","L 1159 3500 \n","L 1159 2956 \n","Q 1341 3275 1631 3429 \n","Q 1922 3584 2338 3584 \n","Q 2397 3584 2469 3576 \n","Q 2541 3569 2628 3553 \n","L 2631 2963 \n","z\n","\" transform=\"scale(0.015625)\"/>\n","       <path id=\"DejaVuSans-61\" d=\"M 2194 1759 \n","Q 1497 1759 1228 1600 \n","Q 959 1441 959 1056 \n","Q 959 750 1161 570 \n","Q 1363 391 1709 391 \n","Q 2188 391 2477 730 \n","Q 2766 1069 2766 1631 \n","L 2766 1759 \n","L 2194 1759 \n","z\n","M 3341 1997 \n","L 3341 0 \n","L 2766 0 \n","L 2766 531 \n","Q 2569 213 2275 61 \n","Q 1981 -91 1556 -91 \n","Q 1019 -91 701 211 \n","Q 384 513 384 1019 \n","Q 384 1609 779 1909 \n","Q 1175 2209 1959 2209 \n","L 2766 2209 \n","L 2766 2266 \n","Q 2766 2663 2505 2880 \n","Q 2244 3097 1772 3097 \n","Q 1472 3097 1187 3025 \n","Q 903 2953 641 2809 \n","L 641 3341 \n","Q 956 3463 1253 3523 \n","Q 1550 3584 1831 3584 \n","Q 2591 3584 2966 3190 \n","Q 3341 2797 3341 1997 \n","z\n","\" transform=\"scale(0.015625)\"/>\n","       <path id=\"DejaVuSans-69\" d=\"M 603 3500 \n","L 1178 3500 \n","L 1178 0 \n","L 603 0 \n","L 603 3500 \n","z\n","M 603 4863 \n","L 1178 4863 \n","L 1178 4134 \n","L 603 4134 \n","L 603 4863 \n","z\n","\" transform=\"scale(0.015625)\"/>\n","       <path id=\"DejaVuSans-6e\" d=\"M 3513 2113 \n","L 3513 0 \n","L 2938 0 \n","L 2938 2094 \n","Q 2938 2591 2744 2837 \n","Q 2550 3084 2163 3084 \n","Q 1697 3084 1428 2787 \n","Q 1159 2491 1159 1978 \n","L 1159 0 \n","L 581 0 \n","L 581 3500 \n","L 1159 3500 \n","L 1159 2956 \n","Q 1366 3272 1645 3428 \n","Q 1925 3584 2291 3584 \n","Q 2894 3584 3203 3211 \n","Q 3513 2838 3513 2113 \n","z\n","\" transform=\"scale(0.015625)\"/>\n","      </defs>\n","      <use xlink:href=\"#DejaVuSans-74\"/>\n","      <use xlink:href=\"#DejaVuSans-72\" x=\"39.208984\"/>\n","      <use xlink:href=\"#DejaVuSans-61\" x=\"80.322266\"/>\n","      <use xlink:href=\"#DejaVuSans-69\" x=\"141.601562\"/>\n","      <use xlink:href=\"#DejaVuSans-6e\" x=\"169.384766\"/>\n","     </g>\n","    </g>\n","    <g id=\"line2d_52\">\n","     <path d=\"M 186.403125 34.976562 \n","L 196.403125 34.976562 \n","L 206.403125 34.976562 \n","\" style=\"fill: none; stroke-dasharray: 5.55,2.4; stroke-dashoffset: 0; stroke: #bf00bf; stroke-width: 1.5\"/>\n","    </g>\n","    <g id=\"text_13\">\n","     <!-- test -->\n","     <g transform=\"translate(214.403125 38.476562)scale(0.1 -0.1)\">\n","      <use xlink:href=\"#DejaVuSans-74\"/>\n","      <use xlink:href=\"#DejaVuSans-65\" x=\"39.208984\"/>\n","      <use xlink:href=\"#DejaVuSans-73\" x=\"100.732422\"/>\n","      <use xlink:href=\"#DejaVuSans-74\" x=\"152.832031\"/>\n","     </g>\n","    </g>\n","   </g>\n","  </g>\n"," </g>\n"," <defs>\n","  <clipPath id=\"p97ac790d64\">\n","   <rect x=\"51.378125\" y=\"7.2\" width=\"195.3\" height=\"135.9\"/>\n","  </clipPath>\n"," </defs>\n","</svg>\n"],"text/plain":["<Figure size 252x180 with 1 Axes>"]},"metadata":{"needs_background":"light"},"output_type":"display_data"}],"source":["train_concise(3)"],"id":"oqZWJbBshB9E"},{"cell_type":"markdown","metadata":{"origin_pos":32,"id":"kC7TD-pXhB9E"},"source":["到目前为止，我们只接触到一个简单线性函数的概念。\n","此外，由什么构成一个简单的非线性函数可能是一个更复杂的问题。\n","例如，[再生核希尔伯特空间（RKHS）](https://en.wikipedia.org/wiki/Reproducing_kernel_Hilbert_space)\n","允许在非线性环境中应用为线性函数引入的工具。\n","不幸的是，基于RKHS的算法往往难以应用到大型、高维的数据。\n","在这本书中，我们将默认使用简单的启发式方法，即在深层网络的所有层上应用权重衰减。\n","\n","## 小结\n","\n","* 正则化是处理过拟合的常用方法：在训练集的损失函数中加入惩罚项，以降低学习到的模型的复杂度。\n","* 保持模型简单的一个特别的选择是使用$L_2$惩罚的权重衰减。这会导致学习算法更新步骤中的权重衰减。\n","* 权重衰减功能在深度学习框架的优化器中提供。\n","* 在同一训练代码实现中，不同的参数集可以有不同的更新行为。\n","\n","## 练习\n","\n","1. 在本节的估计问题中使用$\\lambda$的值进行实验。绘制训练和测试精度关于$\\lambda$的函数。你观察到了什么？\n","1. 使用验证集来找到最佳值$\\lambda$。它真的是最优值吗？这有关系吗？\n","1. 如果我们使用$\\sum_i |w_i|$作为我们选择的惩罚（$L_1$正则化），那么更新方程会是什么样子？\n","1. 我们知道$\\|\\mathbf{w}\\|^2 = \\mathbf{w}^\\top \\mathbf{w}$。你能找到类似的矩阵方程吗（见 :numref:`subsec_lin-algebra-norms` 中的Frobenius范数）？\n","1. 回顾训练误差和泛化误差之间的关系。除了权重衰减、增加训练数据、使用适当复杂度的模型之外，你还能想出其他什么方法来处理过拟合？\n","1. 在贝叶斯统计中，我们使用先验和似然的乘积，通过公式$P(w \\mid x) \\propto P(x \\mid w) P(w)$得到后验。如何得到带正则化的$P(w)$？\n"],"id":"kC7TD-pXhB9E"},{"cell_type":"markdown","metadata":{"origin_pos":34,"tab":["pytorch"],"id":"TnxS-JLLhB9E"},"source":["[Discussions](https://discuss.d2l.ai/t/1808)\n"],"id":"TnxS-JLLhB9E"},{"cell_type":"code","execution_count":null,"id":"7f882393","metadata":{"id":"7f882393"},"outputs":[],"source":["!pip install git+https://github.com/d2l-ai/d2l-zh@release  # installing d2l\n","!pip install git+https://github.com/ipython/matplotlib-inline # installing matplotlib-inline"]},{"cell_type":"markdown","id":"fe9c6c15","metadata":{"origin_pos":0,"id":"fe9c6c15"},"source":["# 多层感知机\n","\n","\n","## 在网络中加入隐藏层\n","\n","我们可以通过在网络中加入一个或多个隐藏层来克服线性模型的限制，\n","使其能处理更普遍的函数关系类型。\n","要做到这一点，最简单的方法是将许多全连接层堆叠在一起。\n","每一层都输出到上面的层，直到生成最后的输出。\n","我们可以把前$L-1$层看作表示，把最后一层看作线性预测器。\n","这种架构通常称为多层感知机（multilayer perceptron），通常缩写为*MLP*。\n","下面，我们以图的方式描述了多层感知机。\n","\n","![一个单隐藏层的多层感知机，具有5个隐藏单元](http://d2l.ai/_images/mlp.svg)\n","\n","\n","这个多层感知机有4个输入，3个输出，其隐藏层包含5个隐藏单元。\n","输入层不涉及任何计算，因此使用此网络产生输出只需要实现隐藏层和输出层的计算。\n","因此，这个多层感知机中的层数为2。\n","注意，这两个层都是全连接的。\n","每个输入都会影响隐藏层中的每个神经元，\n","而隐藏层中的每个神经元又会影响输出层中的每个神经元。\n","\n","然而，\n","具有全连接层的多层感知机的参数开销可能会高得令人望而却步。\n","即使在不改变输入或输出大小的情况下，\n","可能在参数节约和模型有效性之间进行权衡\n","\n","## 从线性到非线性\n","\n","同之前的章节一样，\n","我们通过矩阵$\\mathbf{X} \\in \\mathbb{R}^{n \\times d}$\n","来表示$n$个样本的小批量，\n","其中每个样本具有$d$个输入特征。\n","对于具有$h$个隐藏单元的单隐藏层多层感知机，\n","用$\\mathbf{H} \\in \\mathbb{R}^{n \\times h}$表示隐藏层的输出，\n","称为*隐藏表示*（hidden representations）。\n","在数学或代码中，$\\mathbf{H}$也被称为*隐藏层变量*（hidden-layer variable）\n","或*隐藏变量*（hidden variable）。\n","因为隐藏层和输出层都是全连接的，\n","所以我们有隐藏层权重$\\mathbf{W}^{(1)} \\in \\mathbb{R}^{d \\times h}$\n","和隐藏层偏置$\\mathbf{b}^{(1)} \\in \\mathbb{R}^{1 \\times h}$\n","以及输出层权重$\\mathbf{W}^{(2)} \\in \\mathbb{R}^{h \\times q}$\n","和输出层偏置$\\mathbf{b}^{(2)} \\in \\mathbb{R}^{1 \\times q}$。\n","形式上，我们按如下方式计算单隐藏层多层感知机的输出\n","$\\mathbf{O} \\in \\mathbb{R}^{n \\times q}$：\n","\n","$$\n","\\begin{aligned}\n","    \\mathbf{H} & = \\mathbf{X} \\mathbf{W}^{(1)} + \\mathbf{b}^{(1)}, \\\\\n","    \\mathbf{O} & = \\mathbf{H}\\mathbf{W}^{(2)} + \\mathbf{b}^{(2)}.\n","\\end{aligned}\n","$$\n","\n","注意在添加隐藏层之后，模型现在需要跟踪和更新额外的参数。\n","可我们能从中得到什么好处呢？\n","你可能会惊讶地发现：在上面定义的模型里，我们没有好处！\n","原因很简单：上面的隐藏单元由输入的仿射函数给出，\n","而输出（softmax操作前）只是隐藏单元的仿射函数。\n","仿射函数的仿射函数本身就是仿射函数，\n","但是我们之前的线性模型已经能够表示任何仿射函数。\n","\n","我们可以证明这一等价性，即对于任意权重值，\n","我们只需合并隐藏层，便可产生具有参数\n","$\\mathbf{W} = \\mathbf{W}^{(1)}\\mathbf{W}^{(2)}$\n","和$\\mathbf{b} = \\mathbf{b}^{(1)} \\mathbf{W}^{(2)} + \\mathbf{b}^{(2)}$\n","的等价单层模型：\n","\n","$$\n","\\mathbf{O} = (\\mathbf{X} \\mathbf{W}^{(1)} + \\mathbf{b}^{(1)})\\mathbf{W}^{(2)} + \\mathbf{b}^{(2)} = \\mathbf{X} \\mathbf{W}^{(1)}\\mathbf{W}^{(2)} + \\mathbf{b}^{(1)} \\mathbf{W}^{(2)} + \\mathbf{b}^{(2)} = \\mathbf{X} \\mathbf{W} + \\mathbf{b}.\n","$$\n","\n","为了发挥多层架构的潜力，\n","我们还需要一个额外的关键要素：\n","在仿射变换之后对每个隐藏单元应用非线性的*激活函数*（activation function）$\\sigma$。\n","激活函数的输出（例如，$\\sigma(\\cdot)$）被称为*活性值*（activations）。\n","一般来说，有了激活函数，就不可能再将我们的多层感知机退化成线性模型：\n","\n","$$\n","\\begin{aligned}\n","    \\mathbf{H} & = \\sigma(\\mathbf{X} \\mathbf{W}^{(1)} + \\mathbf{b}^{(1)}), \\\\\n","    \\mathbf{O} & = \\mathbf{H}\\mathbf{W}^{(2)} + \\mathbf{b}^{(2)}.\\\\\n","\\end{aligned}\n","$$\n","\n","由于$\\mathbf{X}$中的每一行对应于小批量中的一个样本，\n","出于记号习惯的考量，\n","我们定义非线性函数$\\sigma$也以按行的方式作用于其输入，\n","即一次计算一个样本。\n","我们在中\n","以相同的方式使用了softmax符号来表示按行操作。\n","但是在本节中，我们应用于隐藏层的激活函数通常不仅按行操作，也按元素操作。\n","这意味着在计算每一层的线性部分之后，我们可以计算每个活性值，\n","而不需要查看其他隐藏单元所取的值。对于大多数激活函数都是这样。\n","\n","为了构建更通用的多层感知机，\n","我们可以继续堆叠这样的隐藏层，\n","例如$\\mathbf{H}^{(1)} = \\sigma_1(\\mathbf{X} \\mathbf{W}^{(1)} + \\mathbf{b}^{(1)})$和$\\mathbf{H}^{(2)} = \\sigma_2(\\mathbf{H}^{(1)} \\mathbf{W}^{(2)} + \\mathbf{b}^{(2)})$，\n","一层叠一层，从而产生更有表达能力的模型。\n","\n","## 通用近似定理\n","\n","多层感知机可以通过隐藏神经元，捕捉到输入之间复杂的相互作用，\n","这些神经元依赖于每个输入的值。\n","我们可以很容易地设计隐藏节点来执行任意计算。\n","例如，在一对输入上进行基本逻辑操作，多层感知机是通用近似器。\n","即使是网络只有一个隐藏层，给定足够的神经元和正确的权重，\n","我们可以对任意函数建模，尽管实际中学习该函数是很困难的。\n","神经网络有点像C语言。\n","C语言和任何其他现代编程语言一样，能够表达任何可计算的程序。\n","但实际上，想出一个符合规范的程序才是最困难的部分。\n","\n","而且，虽然一个单隐层网络能学习任何函数，\n","但并不意味着我们应该尝试使用单隐藏层网络来解决所有问题。\n","事实上，通过使用更深（而不是更广）的网络，我们可以更容易地逼近许多函数。\n","我们将在后面的章节中进行更细致的讨论。\n","\n","## 激活函数\n","\n","激活函数（activation function）通过计算加权和并加上偏置来确定神经元是否应该被激活，\n","它们将输入信号转换为输出的可微运算。\n","大多数激活函数都是非线性的。\n","由于激活函数是深度学习的基础，下面(**简要介绍一些常见的激活函数**)。\n"]},{"cell_type":"code","execution_count":null,"id":"d853f666","metadata":{"execution":{"iopub.execute_input":"2022-07-31T22:40:52.067506Z","iopub.status.busy":"2022-07-31T22:40:52.066933Z","iopub.status.idle":"2022-07-31T22:40:53.952784Z","shell.execute_reply":"2022-07-31T22:40:53.952051Z"},"origin_pos":2,"tab":["pytorch"],"id":"d853f666"},"outputs":[],"source":["%matplotlib inline\n","import torch\n","from d2l import torch as d2l"]},{"cell_type":"markdown","id":"76b73fb7","metadata":{"origin_pos":4,"id":"76b73fb7"},"source":["### ReLU函数\n","\n","最受欢迎的激活函数是*修正线性单元*（Rectified linear unit，*ReLU*），\n","因为它实现简单，同时在各种预测任务中表现良好。\n","[**ReLU提供了一种非常简单的非线性变换**]。\n","给定元素$x$，ReLU函数被定义为该元素与$0$的最大值：\n","\n","(**$$\\operatorname{ReLU}(x) = \\max(x, 0).$$**)\n","\n","通俗地说，ReLU函数通过将相应的活性值设为0，仅保留正元素并丢弃所有负元素。\n","为了直观感受一下，我们可以画出函数的曲线图。\n","正如从图中所看到，激活函数是分段线性的。\n"]},{"cell_type":"code","execution_count":null,"id":"add67469","metadata":{"execution":{"iopub.execute_input":"2022-07-31T22:40:53.956473Z","iopub.status.busy":"2022-07-31T22:40:53.956078Z","iopub.status.idle":"2022-07-31T22:40:54.172064Z","shell.execute_reply":"2022-07-31T22:40:54.171405Z"},"origin_pos":6,"tab":["pytorch"],"id":"add67469","outputId":"917bd120-9a89-4fa6-ede6-e78eb91e35f9","colab":{"base_uri":"https://localhost:8080/","height":262},"executionInfo":{"status":"ok","timestamp":1663765165950,"user_tz":-480,"elapsed":649,"user":{"displayName":"Geeks Z","userId":"18417645384289412694"}}},"outputs":[{"output_type":"display_data","data":{"text/plain":["<Figure size 360x180 with 1 Axes>"],"image/svg+xml":"<?xml version=\"1.0\" encoding=\"utf-8\" standalone=\"no\"?>\n<!DOCTYPE svg PUBLIC \"-//W3C//DTD SVG 1.1//EN\"\n  \"http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd\">\n<svg xmlns:xlink=\"http://www.w3.org/1999/xlink\" width=\"320.440625pt\" height=\"180.65625pt\" viewBox=\"0 0 320.440625 180.65625\" xmlns=\"http://www.w3.org/2000/svg\" version=\"1.1\">\n <metadata>\n  <rdf:RDF xmlns:dc=\"http://purl.org/dc/elements/1.1/\" xmlns:cc=\"http://creativecommons.org/ns#\" xmlns:rdf=\"http://www.w3.org/1999/02/22-rdf-syntax-ns#\">\n   <cc:Work>\n    <dc:type rdf:resource=\"http://purl.org/dc/dcmitype/StillImage\"/>\n    <dc:date>2022-09-21T12:59:25.751231</dc:date>\n    <dc:format>image/svg+xml</dc:format>\n    <dc:creator>\n     <cc:Agent>\n      <dc:title>Matplotlib v3.5.1, https://matplotlib.org/</dc:title>\n     </cc:Agent>\n    </dc:creator>\n   </cc:Work>\n  </rdf:RDF>\n </metadata>\n <defs>\n  <style type=\"text/css\">*{stroke-linejoin: round; stroke-linecap: butt}</style>\n </defs>\n <g id=\"figure_1\">\n  <g id=\"patch_1\">\n   <path d=\"M 0 180.65625 \nL 320.440625 180.65625 \nL 320.440625 0 \nL 0 0 \nL 0 180.65625 \nz\n\" style=\"fill: none\"/>\n  </g>\n  <g id=\"axes_1\">\n   <g id=\"patch_2\">\n    <path d=\"M 34.240625 143.1 \nL 313.240625 143.1 \nL 313.240625 7.2 \nL 34.240625 7.2 \nz\n\" style=\"fill: #ffffff\"/>\n   </g>\n   <g id=\"matplotlib.axis_1\">\n    <g id=\"xtick_1\">\n     <g id=\"line2d_1\">\n      <path d=\"M 46.922443 143.1 \nL 46.922443 7.2 \n\" clip-path=\"url(#pd166ea5b6a)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n     </g>\n     <g id=\"line2d_2\">\n      <defs>\n       <path id=\"m69db01a52d\" d=\"M 0 0 \nL 0 3.5 \n\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n      </defs>\n      <g>\n       <use xlink:href=\"#m69db01a52d\" x=\"46.922443\" y=\"143.1\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n      </g>\n     </g>\n     <g id=\"text_1\">\n      <!-- −8 -->\n      <g transform=\"translate(39.551349 157.698438)scale(0.1 -0.1)\">\n       <defs>\n        <path id=\"DejaVuSans-2212\" d=\"M 678 2272 \nL 4684 2272 \nL 4684 1741 \nL 678 1741 \nL 678 2272 \nz\n\" transform=\"scale(0.015625)\"/>\n        <path id=\"DejaVuSans-38\" d=\"M 2034 2216 \nQ 1584 2216 1326 1975 \nQ 1069 1734 1069 1313 \nQ 1069 891 1326 650 \nQ 1584 409 2034 409 \nQ 2484 409 2743 651 \nQ 3003 894 3003 1313 \nQ 3003 1734 2745 1975 \nQ 2488 2216 2034 2216 \nz\nM 1403 2484 \nQ 997 2584 770 2862 \nQ 544 3141 544 3541 \nQ 544 4100 942 4425 \nQ 1341 4750 2034 4750 \nQ 2731 4750 3128 4425 \nQ 3525 4100 3525 3541 \nQ 3525 3141 3298 2862 \nQ 3072 2584 2669 2484 \nQ 3125 2378 3379 2068 \nQ 3634 1759 3634 1313 \nQ 3634 634 3220 271 \nQ 2806 -91 2034 -91 \nQ 1263 -91 848 271 \nQ 434 634 434 1313 \nQ 434 1759 690 2068 \nQ 947 2378 1403 2484 \nz\nM 1172 3481 \nQ 1172 3119 1398 2916 \nQ 1625 2713 2034 2713 \nQ 2441 2713 2670 2916 \nQ 2900 3119 2900 3481 \nQ 2900 3844 2670 4047 \nQ 2441 4250 2034 4250 \nQ 1625 4250 1398 4047 \nQ 1172 3844 1172 3481 \nz\n\" transform=\"scale(0.015625)\"/>\n       </defs>\n       <use xlink:href=\"#DejaVuSans-2212\"/>\n       <use xlink:href=\"#DejaVuSans-38\" x=\"83.789062\"/>\n      </g>\n     </g>\n    </g>\n    <g id=\"xtick_2\">\n     <g id=\"line2d_3\">\n      <path d=\"M 78.826389 143.1 \nL 78.826389 7.2 \n\" clip-path=\"url(#pd166ea5b6a)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n     </g>\n     <g id=\"line2d_4\">\n      <g>\n       <use xlink:href=\"#m69db01a52d\" x=\"78.826389\" y=\"143.1\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n      </g>\n     </g>\n     <g id=\"text_2\">\n      <!-- −6 -->\n      <g transform=\"translate(71.455295 157.698438)scale(0.1 -0.1)\">\n       <defs>\n        <path id=\"DejaVuSans-36\" d=\"M 2113 2584 \nQ 1688 2584 1439 2293 \nQ 1191 2003 1191 1497 \nQ 1191 994 1439 701 \nQ 1688 409 2113 409 \nQ 2538 409 2786 701 \nQ 3034 994 3034 1497 \nQ 3034 2003 2786 2293 \nQ 2538 2584 2113 2584 \nz\nM 3366 4563 \nL 3366 3988 \nQ 3128 4100 2886 4159 \nQ 2644 4219 2406 4219 \nQ 1781 4219 1451 3797 \nQ 1122 3375 1075 2522 \nQ 1259 2794 1537 2939 \nQ 1816 3084 2150 3084 \nQ 2853 3084 3261 2657 \nQ 3669 2231 3669 1497 \nQ 3669 778 3244 343 \nQ 2819 -91 2113 -91 \nQ 1303 -91 875 529 \nQ 447 1150 447 2328 \nQ 447 3434 972 4092 \nQ 1497 4750 2381 4750 \nQ 2619 4750 2861 4703 \nQ 3103 4656 3366 4563 \nz\n\" transform=\"scale(0.015625)\"/>\n       </defs>\n       <use xlink:href=\"#DejaVuSans-2212\"/>\n       <use xlink:href=\"#DejaVuSans-36\" x=\"83.789062\"/>\n      </g>\n     </g>\n    </g>\n    <g id=\"xtick_3\">\n     <g id=\"line2d_5\">\n      <path d=\"M 110.730335 143.1 \nL 110.730335 7.2 \n\" clip-path=\"url(#pd166ea5b6a)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n     </g>\n     <g id=\"line2d_6\">\n      <g>\n       <use xlink:href=\"#m69db01a52d\" x=\"110.730335\" y=\"143.1\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n      </g>\n     </g>\n     <g id=\"text_3\">\n      <!-- −4 -->\n      <g transform=\"translate(103.359241 157.698438)scale(0.1 -0.1)\">\n       <defs>\n        <path id=\"DejaVuSans-34\" d=\"M 2419 4116 \nL 825 1625 \nL 2419 1625 \nL 2419 4116 \nz\nM 2253 4666 \nL 3047 4666 \nL 3047 1625 \nL 3713 1625 \nL 3713 1100 \nL 3047 1100 \nL 3047 0 \nL 2419 0 \nL 2419 1100 \nL 313 1100 \nL 313 1709 \nL 2253 4666 \nz\n\" transform=\"scale(0.015625)\"/>\n       </defs>\n       <use xlink:href=\"#DejaVuSans-2212\"/>\n       <use xlink:href=\"#DejaVuSans-34\" x=\"83.789062\"/>\n      </g>\n     </g>\n    </g>\n    <g id=\"xtick_4\">\n     <g id=\"line2d_7\">\n      <path d=\"M 142.634281 143.1 \nL 142.634281 7.2 \n\" clip-path=\"url(#pd166ea5b6a)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n     </g>\n     <g id=\"line2d_8\">\n      <g>\n       <use xlink:href=\"#m69db01a52d\" x=\"142.634281\" y=\"143.1\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n      </g>\n     </g>\n     <g id=\"text_4\">\n      <!-- −2 -->\n      <g transform=\"translate(135.263187 157.698438)scale(0.1 -0.1)\">\n       <defs>\n        <path id=\"DejaVuSans-32\" d=\"M 1228 531 \nL 3431 531 \nL 3431 0 \nL 469 0 \nL 469 531 \nQ 828 903 1448 1529 \nQ 2069 2156 2228 2338 \nQ 2531 2678 2651 2914 \nQ 2772 3150 2772 3378 \nQ 2772 3750 2511 3984 \nQ 2250 4219 1831 4219 \nQ 1534 4219 1204 4116 \nQ 875 4013 500 3803 \nL 500 4441 \nQ 881 4594 1212 4672 \nQ 1544 4750 1819 4750 \nQ 2544 4750 2975 4387 \nQ 3406 4025 3406 3419 \nQ 3406 3131 3298 2873 \nQ 3191 2616 2906 2266 \nQ 2828 2175 2409 1742 \nQ 1991 1309 1228 531 \nz\n\" transform=\"scale(0.015625)\"/>\n       </defs>\n       <use xlink:href=\"#DejaVuSans-2212\"/>\n       <use xlink:href=\"#DejaVuSans-32\" x=\"83.789062\"/>\n      </g>\n     </g>\n    </g>\n    <g id=\"xtick_5\">\n     <g id=\"line2d_9\">\n      <path d=\"M 174.538227 143.1 \nL 174.538227 7.2 \n\" clip-path=\"url(#pd166ea5b6a)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n     </g>\n     <g id=\"line2d_10\">\n      <g>\n       <use xlink:href=\"#m69db01a52d\" x=\"174.538227\" y=\"143.1\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n      </g>\n     </g>\n     <g id=\"text_5\">\n      <!-- 0 -->\n      <g transform=\"translate(171.356977 157.698438)scale(0.1 -0.1)\">\n       <defs>\n        <path id=\"DejaVuSans-30\" d=\"M 2034 4250 \nQ 1547 4250 1301 3770 \nQ 1056 3291 1056 2328 \nQ 1056 1369 1301 889 \nQ 1547 409 2034 409 \nQ 2525 409 2770 889 \nQ 3016 1369 3016 2328 \nQ 3016 3291 2770 3770 \nQ 2525 4250 2034 4250 \nz\nM 2034 4750 \nQ 2819 4750 3233 4129 \nQ 3647 3509 3647 2328 \nQ 3647 1150 3233 529 \nQ 2819 -91 2034 -91 \nQ 1250 -91 836 529 \nQ 422 1150 422 2328 \nQ 422 3509 836 4129 \nQ 1250 4750 2034 4750 \nz\n\" transform=\"scale(0.015625)\"/>\n       </defs>\n       <use xlink:href=\"#DejaVuSans-30\"/>\n      </g>\n     </g>\n    </g>\n    <g id=\"xtick_6\">\n     <g id=\"line2d_11\">\n      <path d=\"M 206.442173 143.1 \nL 206.442173 7.2 \n\" clip-path=\"url(#pd166ea5b6a)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n     </g>\n     <g id=\"line2d_12\">\n      <g>\n       <use xlink:href=\"#m69db01a52d\" x=\"206.442173\" y=\"143.1\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n      </g>\n     </g>\n     <g id=\"text_6\">\n      <!-- 2 -->\n      <g transform=\"translate(203.260923 157.698438)scale(0.1 -0.1)\">\n       <use xlink:href=\"#DejaVuSans-32\"/>\n      </g>\n     </g>\n    </g>\n    <g id=\"xtick_7\">\n     <g id=\"line2d_13\">\n      <path d=\"M 238.346118 143.1 \nL 238.346118 7.2 \n\" clip-path=\"url(#pd166ea5b6a)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n     </g>\n     <g id=\"line2d_14\">\n      <g>\n       <use xlink:href=\"#m69db01a52d\" x=\"238.346118\" y=\"143.1\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n      </g>\n     </g>\n     <g id=\"text_7\">\n      <!-- 4 -->\n      <g transform=\"translate(235.164868 157.698438)scale(0.1 -0.1)\">\n       <use xlink:href=\"#DejaVuSans-34\"/>\n      </g>\n     </g>\n    </g>\n    <g id=\"xtick_8\">\n     <g id=\"line2d_15\">\n      <path d=\"M 270.250064 143.1 \nL 270.250064 7.2 \n\" clip-path=\"url(#pd166ea5b6a)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n     </g>\n     <g id=\"line2d_16\">\n      <g>\n       <use xlink:href=\"#m69db01a52d\" x=\"270.250064\" y=\"143.1\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n      </g>\n     </g>\n     <g id=\"text_8\">\n      <!-- 6 -->\n      <g transform=\"translate(267.068814 157.698438)scale(0.1 -0.1)\">\n       <use xlink:href=\"#DejaVuSans-36\"/>\n      </g>\n     </g>\n    </g>\n    <g id=\"xtick_9\">\n     <g id=\"line2d_17\">\n      <path d=\"M 302.15401 143.1 \nL 302.15401 7.2 \n\" clip-path=\"url(#pd166ea5b6a)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n     </g>\n     <g id=\"line2d_18\">\n      <g>\n       <use xlink:href=\"#m69db01a52d\" x=\"302.15401\" y=\"143.1\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n      </g>\n     </g>\n     <g id=\"text_9\">\n      <!-- 8 -->\n      <g transform=\"translate(298.97276 157.698438)scale(0.1 -0.1)\">\n       <use xlink:href=\"#DejaVuSans-38\"/>\n      </g>\n     </g>\n    </g>\n    <g id=\"text_10\">\n     <!-- x -->\n     <g transform=\"translate(170.78125 171.376563)scale(0.1 -0.1)\">\n      <defs>\n       <path id=\"DejaVuSans-78\" d=\"M 3513 3500 \nL 2247 1797 \nL 3578 0 \nL 2900 0 \nL 1881 1375 \nL 863 0 \nL 184 0 \nL 1544 1831 \nL 300 3500 \nL 978 3500 \nL 1906 2253 \nL 2834 3500 \nL 3513 3500 \nz\n\" transform=\"scale(0.015625)\"/>\n      </defs>\n      <use xlink:href=\"#DejaVuSans-78\"/>\n     </g>\n    </g>\n   </g>\n   <g id=\"matplotlib.axis_2\">\n    <g id=\"ytick_1\">\n     <g id=\"line2d_19\">\n      <path d=\"M 34.240625 136.922727 \nL 313.240625 136.922727 \n\" clip-path=\"url(#pd166ea5b6a)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n     </g>\n     <g id=\"line2d_20\">\n      <defs>\n       <path id=\"m4482fb8504\" d=\"M 0 0 \nL -3.5 0 \n\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n      </defs>\n      <g>\n       <use xlink:href=\"#m4482fb8504\" x=\"34.240625\" y=\"136.922727\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n      </g>\n     </g>\n     <g id=\"text_11\">\n      <!-- 0 -->\n      <g transform=\"translate(20.878125 140.721946)scale(0.1 -0.1)\">\n       <use xlink:href=\"#DejaVuSans-30\"/>\n      </g>\n     </g>\n    </g>\n    <g id=\"ytick_2\">\n     <g id=\"line2d_21\">\n      <path d=\"M 34.240625 105.645395 \nL 313.240625 105.645395 \n\" clip-path=\"url(#pd166ea5b6a)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n     </g>\n     <g id=\"line2d_22\">\n      <g>\n       <use xlink:href=\"#m4482fb8504\" x=\"34.240625\" y=\"105.645395\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n      </g>\n     </g>\n     <g id=\"text_12\">\n      <!-- 2 -->\n      <g transform=\"translate(20.878125 109.444614)scale(0.1 -0.1)\">\n       <use xlink:href=\"#DejaVuSans-32\"/>\n      </g>\n     </g>\n    </g>\n    <g id=\"ytick_3\">\n     <g id=\"line2d_23\">\n      <path d=\"M 34.240625 74.368064 \nL 313.240625 74.368064 \n\" clip-path=\"url(#pd166ea5b6a)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n     </g>\n     <g id=\"line2d_24\">\n      <g>\n       <use xlink:href=\"#m4482fb8504\" x=\"34.240625\" y=\"74.368064\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n      </g>\n     </g>\n     <g id=\"text_13\">\n      <!-- 4 -->\n      <g transform=\"translate(20.878125 78.167282)scale(0.1 -0.1)\">\n       <use xlink:href=\"#DejaVuSans-34\"/>\n      </g>\n     </g>\n    </g>\n    <g id=\"ytick_4\">\n     <g id=\"line2d_25\">\n      <path d=\"M 34.240625 43.090732 \nL 313.240625 43.090732 \n\" clip-path=\"url(#pd166ea5b6a)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n     </g>\n     <g id=\"line2d_26\">\n      <g>\n       <use xlink:href=\"#m4482fb8504\" x=\"34.240625\" y=\"43.090732\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n      </g>\n     </g>\n     <g id=\"text_14\">\n      <!-- 6 -->\n      <g transform=\"translate(20.878125 46.889951)scale(0.1 -0.1)\">\n       <use xlink:href=\"#DejaVuSans-36\"/>\n      </g>\n     </g>\n    </g>\n    <g id=\"ytick_5\">\n     <g id=\"line2d_27\">\n      <path d=\"M 34.240625 11.8134 \nL 313.240625 11.8134 \n\" clip-path=\"url(#pd166ea5b6a)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n     </g>\n     <g id=\"line2d_28\">\n      <g>\n       <use xlink:href=\"#m4482fb8504\" x=\"34.240625\" y=\"11.8134\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n      </g>\n     </g>\n     <g id=\"text_15\">\n      <!-- 8 -->\n      <g transform=\"translate(20.878125 15.612619)scale(0.1 -0.1)\">\n       <use xlink:href=\"#DejaVuSans-38\"/>\n      </g>\n     </g>\n    </g>\n    <g id=\"text_16\">\n     <!-- relu(x) -->\n     <g transform=\"translate(14.798438 91.588281)rotate(-90)scale(0.1 -0.1)\">\n      <defs>\n       <path id=\"DejaVuSans-72\" d=\"M 2631 2963 \nQ 2534 3019 2420 3045 \nQ 2306 3072 2169 3072 \nQ 1681 3072 1420 2755 \nQ 1159 2438 1159 1844 \nL 1159 0 \nL 581 0 \nL 581 3500 \nL 1159 3500 \nL 1159 2956 \nQ 1341 3275 1631 3429 \nQ 1922 3584 2338 3584 \nQ 2397 3584 2469 3576 \nQ 2541 3569 2628 3553 \nL 2631 2963 \nz\n\" transform=\"scale(0.015625)\"/>\n       <path id=\"DejaVuSans-65\" d=\"M 3597 1894 \nL 3597 1613 \nL 953 1613 \nQ 991 1019 1311 708 \nQ 1631 397 2203 397 \nQ 2534 397 2845 478 \nQ 3156 559 3463 722 \nL 3463 178 \nQ 3153 47 2828 -22 \nQ 2503 -91 2169 -91 \nQ 1331 -91 842 396 \nQ 353 884 353 1716 \nQ 353 2575 817 3079 \nQ 1281 3584 2069 3584 \nQ 2775 3584 3186 3129 \nQ 3597 2675 3597 1894 \nz\nM 3022 2063 \nQ 3016 2534 2758 2815 \nQ 2500 3097 2075 3097 \nQ 1594 3097 1305 2825 \nQ 1016 2553 972 2059 \nL 3022 2063 \nz\n\" transform=\"scale(0.015625)\"/>\n       <path id=\"DejaVuSans-6c\" d=\"M 603 4863 \nL 1178 4863 \nL 1178 0 \nL 603 0 \nL 603 4863 \nz\n\" transform=\"scale(0.015625)\"/>\n       <path id=\"DejaVuSans-75\" d=\"M 544 1381 \nL 544 3500 \nL 1119 3500 \nL 1119 1403 \nQ 1119 906 1312 657 \nQ 1506 409 1894 409 \nQ 2359 409 2629 706 \nQ 2900 1003 2900 1516 \nL 2900 3500 \nL 3475 3500 \nL 3475 0 \nL 2900 0 \nL 2900 538 \nQ 2691 219 2414 64 \nQ 2138 -91 1772 -91 \nQ 1169 -91 856 284 \nQ 544 659 544 1381 \nz\nM 1991 3584 \nL 1991 3584 \nz\n\" transform=\"scale(0.015625)\"/>\n       <path id=\"DejaVuSans-28\" d=\"M 1984 4856 \nQ 1566 4138 1362 3434 \nQ 1159 2731 1159 2009 \nQ 1159 1288 1364 580 \nQ 1569 -128 1984 -844 \nL 1484 -844 \nQ 1016 -109 783 600 \nQ 550 1309 550 2009 \nQ 550 2706 781 3412 \nQ 1013 4119 1484 4856 \nL 1984 4856 \nz\n\" transform=\"scale(0.015625)\"/>\n       <path id=\"DejaVuSans-29\" d=\"M 513 4856 \nL 1013 4856 \nQ 1481 4119 1714 3412 \nQ 1947 2706 1947 2009 \nQ 1947 1309 1714 600 \nQ 1481 -109 1013 -844 \nL 513 -844 \nQ 928 -128 1133 580 \nQ 1338 1288 1338 2009 \nQ 1338 2731 1133 3434 \nQ 928 4138 513 4856 \nz\n\" transform=\"scale(0.015625)\"/>\n      </defs>\n      <use xlink:href=\"#DejaVuSans-72\"/>\n      <use xlink:href=\"#DejaVuSans-65\" x=\"38.863281\"/>\n      <use xlink:href=\"#DejaVuSans-6c\" x=\"100.386719\"/>\n      <use xlink:href=\"#DejaVuSans-75\" x=\"128.169922\"/>\n      <use xlink:href=\"#DejaVuSans-28\" x=\"191.548828\"/>\n      <use xlink:href=\"#DejaVuSans-78\" x=\"230.5625\"/>\n      <use xlink:href=\"#DejaVuSans-29\" x=\"289.742188\"/>\n     </g>\n    </g>\n   </g>\n   <g id=\"line2d_29\">\n    <path d=\"M 46.922443 136.922727 \nL 174.538227 136.922727 \nL 300.558807 13.377273 \nL 300.558807 13.377273 \n\" clip-path=\"url(#pd166ea5b6a)\" style=\"fill: none; stroke: #1f77b4; stroke-width: 1.5; stroke-linecap: square\"/>\n   </g>\n   <g id=\"patch_3\">\n    <path d=\"M 34.240625 143.1 \nL 34.240625 7.2 \n\" style=\"fill: none; stroke: #000000; stroke-width: 0.8; stroke-linejoin: miter; stroke-linecap: square\"/>\n   </g>\n   <g id=\"patch_4\">\n    <path d=\"M 313.240625 143.1 \nL 313.240625 7.2 \n\" style=\"fill: none; stroke: #000000; stroke-width: 0.8; stroke-linejoin: miter; stroke-linecap: square\"/>\n   </g>\n   <g id=\"patch_5\">\n    <path d=\"M 34.240625 143.1 \nL 313.240625 143.1 \n\" style=\"fill: none; stroke: #000000; stroke-width: 0.8; stroke-linejoin: miter; stroke-linecap: square\"/>\n   </g>\n   <g id=\"patch_6\">\n    <path d=\"M 34.240625 7.2 \nL 313.240625 7.2 \n\" style=\"fill: none; stroke: #000000; stroke-width: 0.8; stroke-linejoin: miter; stroke-linecap: square\"/>\n   </g>\n  </g>\n </g>\n <defs>\n  <clipPath id=\"pd166ea5b6a\">\n   <rect x=\"34.240625\" y=\"7.2\" width=\"279\" height=\"135.9\"/>\n  </clipPath>\n </defs>\n</svg>\n"},"metadata":{"needs_background":"light"}}],"source":["x = torch.arange(-8.0, 8.0, 0.1, requires_grad=True)\n","y = torch.relu(x)\n","d2l.plot(x.detach(), y.detach(), 'x', 'relu(x)', figsize=(5, 2.5))"]},{"cell_type":"markdown","id":"57ca4572","metadata":{"origin_pos":8,"id":"57ca4572"},"source":["当输入为负时，ReLU函数的导数为0，而当输入为正时，ReLU函数的导数为1。\n","注意，当输入值精确等于0时，ReLU函数不可导。\n","在此时，我们默认使用左侧的导数，即当输入为0时导数为0。\n","我们可以忽略这种情况，因为输入可能永远都不会是0。\n","这里引用一句古老的谚语，“如果微妙的边界条件很重要，我们很可能是在研究数学而非工程”，\n","这个观点正好适用于这里。\n","下面我们绘制ReLU函数的导数。\n"]},{"cell_type":"code","execution_count":null,"id":"da421438","metadata":{"execution":{"iopub.execute_input":"2022-07-31T22:40:54.176031Z","iopub.status.busy":"2022-07-31T22:40:54.175539Z","iopub.status.idle":"2022-07-31T22:40:54.378290Z","shell.execute_reply":"2022-07-31T22:40:54.377652Z"},"origin_pos":10,"tab":["pytorch"],"id":"da421438","outputId":"1cc1082b-85d4-4f32-c769-06e0e5dce41f"},"outputs":[{"data":{"image/svg+xml":["<?xml version=\"1.0\" encoding=\"utf-8\" standalone=\"no\"?>\n","<!DOCTYPE svg PUBLIC \"-//W3C//DTD SVG 1.1//EN\"\n","  \"http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd\">\n","<svg xmlns:xlink=\"http://www.w3.org/1999/xlink\" width=\"329.98125pt\" height=\"180.65625pt\" viewBox=\"0 0 329.98125 180.65625\" xmlns=\"http://www.w3.org/2000/svg\" version=\"1.1\">\n"," <metadata>\n","  <rdf:RDF xmlns:dc=\"http://purl.org/dc/elements/1.1/\" xmlns:cc=\"http://creativecommons.org/ns#\" xmlns:rdf=\"http://www.w3.org/1999/02/22-rdf-syntax-ns#\">\n","   <cc:Work>\n","    <dc:type rdf:resource=\"http://purl.org/dc/dcmitype/StillImage\"/>\n","    <dc:date>2022-07-31T22:40:54.334440</dc:date>\n","    <dc:format>image/svg+xml</dc:format>\n","    <dc:creator>\n","     <cc:Agent>\n","      <dc:title>Matplotlib v3.5.1, https://matplotlib.org/</dc:title>\n","     </cc:Agent>\n","    </dc:creator>\n","   </cc:Work>\n","  </rdf:RDF>\n"," </metadata>\n"," <defs>\n","  <style type=\"text/css\">*{stroke-linejoin: round; stroke-linecap: butt}</style>\n"," </defs>\n"," <g id=\"figure_1\">\n","  <g id=\"patch_1\">\n","   <path d=\"M 0 180.65625 \n","L 329.98125 180.65625 \n","L 329.98125 0 \n","L 0 0 \n","L 0 180.65625 \n","z\n","\" style=\"fill: none\"/>\n","  </g>\n","  <g id=\"axes_1\">\n","   <g id=\"patch_2\">\n","    <path d=\"M 43.78125 143.1 \n","L 322.78125 143.1 \n","L 322.78125 7.2 \n","L 43.78125 7.2 \n","z\n","\" style=\"fill: #ffffff\"/>\n","   </g>\n","   <g id=\"matplotlib.axis_1\">\n","    <g id=\"xtick_1\">\n","     <g id=\"line2d_1\">\n","      <path d=\"M 56.463068 143.1 \n","L 56.463068 7.2 \n","\" clip-path=\"url(#p8fd8832baa)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n","     </g>\n","     <g id=\"line2d_2\">\n","      <defs>\n","       <path id=\"m86a7e518fd\" d=\"M 0 0 \n","L 0 3.5 \n","\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n","      </defs>\n","      <g>\n","       <use xlink:href=\"#m86a7e518fd\" x=\"56.463068\" y=\"143.1\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n","      </g>\n","     </g>\n","     <g id=\"text_1\">\n","      <!-- −8 -->\n","      <g transform=\"translate(49.091974 157.698438)scale(0.1 -0.1)\">\n","       <defs>\n","        <path id=\"DejaVuSans-2212\" d=\"M 678 2272 \n","L 4684 2272 \n","L 4684 1741 \n","L 678 1741 \n","L 678 2272 \n","z\n","\" transform=\"scale(0.015625)\"/>\n","        <path id=\"DejaVuSans-38\" d=\"M 2034 2216 \n","Q 1584 2216 1326 1975 \n","Q 1069 1734 1069 1313 \n","Q 1069 891 1326 650 \n","Q 1584 409 2034 409 \n","Q 2484 409 2743 651 \n","Q 3003 894 3003 1313 \n","Q 3003 1734 2745 1975 \n","Q 2488 2216 2034 2216 \n","z\n","M 1403 2484 \n","Q 997 2584 770 2862 \n","Q 544 3141 544 3541 \n","Q 544 4100 942 4425 \n","Q 1341 4750 2034 4750 \n","Q 2731 4750 3128 4425 \n","Q 3525 4100 3525 3541 \n","Q 3525 3141 3298 2862 \n","Q 3072 2584 2669 2484 \n","Q 3125 2378 3379 2068 \n","Q 3634 1759 3634 1313 \n","Q 3634 634 3220 271 \n","Q 2806 -91 2034 -91 \n","Q 1263 -91 848 271 \n","Q 434 634 434 1313 \n","Q 434 1759 690 2068 \n","Q 947 2378 1403 2484 \n","z\n","M 1172 3481 \n","Q 1172 3119 1398 2916 \n","Q 1625 2713 2034 2713 \n","Q 2441 2713 2670 2916 \n","Q 2900 3119 2900 3481 \n","Q 2900 3844 2670 4047 \n","Q 2441 4250 2034 4250 \n","Q 1625 4250 1398 4047 \n","Q 1172 3844 1172 3481 \n","z\n","\" transform=\"scale(0.015625)\"/>\n","       </defs>\n","       <use xlink:href=\"#DejaVuSans-2212\"/>\n","       <use xlink:href=\"#DejaVuSans-38\" x=\"83.789062\"/>\n","      </g>\n","     </g>\n","    </g>\n","    <g id=\"xtick_2\">\n","     <g id=\"line2d_3\">\n","      <path d=\"M 88.367014 143.1 \n","L 88.367014 7.2 \n","\" clip-path=\"url(#p8fd8832baa)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n","     </g>\n","     <g id=\"line2d_4\">\n","      <g>\n","       <use xlink:href=\"#m86a7e518fd\" x=\"88.367014\" y=\"143.1\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n","      </g>\n","     </g>\n","     <g id=\"text_2\">\n","      <!-- −6 -->\n","      <g transform=\"translate(80.99592 157.698438)scale(0.1 -0.1)\">\n","       <defs>\n","        <path id=\"DejaVuSans-36\" d=\"M 2113 2584 \n","Q 1688 2584 1439 2293 \n","Q 1191 2003 1191 1497 \n","Q 1191 994 1439 701 \n","Q 1688 409 2113 409 \n","Q 2538 409 2786 701 \n","Q 3034 994 3034 1497 \n","Q 3034 2003 2786 2293 \n","Q 2538 2584 2113 2584 \n","z\n","M 3366 4563 \n","L 3366 3988 \n","Q 3128 4100 2886 4159 \n","Q 2644 4219 2406 4219 \n","Q 1781 4219 1451 3797 \n","Q 1122 3375 1075 2522 \n","Q 1259 2794 1537 2939 \n","Q 1816 3084 2150 3084 \n","Q 2853 3084 3261 2657 \n","Q 3669 2231 3669 1497 \n","Q 3669 778 3244 343 \n","Q 2819 -91 2113 -91 \n","Q 1303 -91 875 529 \n","Q 447 1150 447 2328 \n","Q 447 3434 972 4092 \n","Q 1497 4750 2381 4750 \n","Q 2619 4750 2861 4703 \n","Q 3103 4656 3366 4563 \n","z\n","\" transform=\"scale(0.015625)\"/>\n","       </defs>\n","       <use xlink:href=\"#DejaVuSans-2212\"/>\n","       <use xlink:href=\"#DejaVuSans-36\" x=\"83.789062\"/>\n","      </g>\n","     </g>\n","    </g>\n","    <g id=\"xtick_3\">\n","     <g id=\"line2d_5\">\n","      <path d=\"M 120.27096 143.1 \n","L 120.27096 7.2 \n","\" clip-path=\"url(#p8fd8832baa)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n","     </g>\n","     <g id=\"line2d_6\">\n","      <g>\n","       <use xlink:href=\"#m86a7e518fd\" x=\"120.27096\" y=\"143.1\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n","      </g>\n","     </g>\n","     <g id=\"text_3\">\n","      <!-- −4 -->\n","      <g transform=\"translate(112.899866 157.698438)scale(0.1 -0.1)\">\n","       <defs>\n","        <path id=\"DejaVuSans-34\" d=\"M 2419 4116 \n","L 825 1625 \n","L 2419 1625 \n","L 2419 4116 \n","z\n","M 2253 4666 \n","L 3047 4666 \n","L 3047 1625 \n","L 3713 1625 \n","L 3713 1100 \n","L 3047 1100 \n","L 3047 0 \n","L 2419 0 \n","L 2419 1100 \n","L 313 1100 \n","L 313 1709 \n","L 2253 4666 \n","z\n","\" transform=\"scale(0.015625)\"/>\n","       </defs>\n","       <use xlink:href=\"#DejaVuSans-2212\"/>\n","       <use xlink:href=\"#DejaVuSans-34\" x=\"83.789062\"/>\n","      </g>\n","     </g>\n","    </g>\n","    <g id=\"xtick_4\">\n","     <g id=\"line2d_7\">\n","      <path d=\"M 152.174906 143.1 \n","L 152.174906 7.2 \n","\" clip-path=\"url(#p8fd8832baa)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n","     </g>\n","     <g id=\"line2d_8\">\n","      <g>\n","       <use xlink:href=\"#m86a7e518fd\" x=\"152.174906\" y=\"143.1\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n","      </g>\n","     </g>\n","     <g id=\"text_4\">\n","      <!-- −2 -->\n","      <g transform=\"translate(144.803812 157.698438)scale(0.1 -0.1)\">\n","       <defs>\n","        <path id=\"DejaVuSans-32\" d=\"M 1228 531 \n","L 3431 531 \n","L 3431 0 \n","L 469 0 \n","L 469 531 \n","Q 828 903 1448 1529 \n","Q 2069 2156 2228 2338 \n","Q 2531 2678 2651 2914 \n","Q 2772 3150 2772 3378 \n","Q 2772 3750 2511 3984 \n","Q 2250 4219 1831 4219 \n","Q 1534 4219 1204 4116 \n","Q 875 4013 500 3803 \n","L 500 4441 \n","Q 881 4594 1212 4672 \n","Q 1544 4750 1819 4750 \n","Q 2544 4750 2975 4387 \n","Q 3406 4025 3406 3419 \n","Q 3406 3131 3298 2873 \n","Q 3191 2616 2906 2266 \n","Q 2828 2175 2409 1742 \n","Q 1991 1309 1228 531 \n","z\n","\" transform=\"scale(0.015625)\"/>\n","       </defs>\n","       <use xlink:href=\"#DejaVuSans-2212\"/>\n","       <use xlink:href=\"#DejaVuSans-32\" x=\"83.789062\"/>\n","      </g>\n","     </g>\n","    </g>\n","    <g id=\"xtick_5\">\n","     <g id=\"line2d_9\">\n","      <path d=\"M 184.078852 143.1 \n","L 184.078852 7.2 \n","\" clip-path=\"url(#p8fd8832baa)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n","     </g>\n","     <g id=\"line2d_10\">\n","      <g>\n","       <use xlink:href=\"#m86a7e518fd\" x=\"184.078852\" y=\"143.1\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n","      </g>\n","     </g>\n","     <g id=\"text_5\">\n","      <!-- 0 -->\n","      <g transform=\"translate(180.897602 157.698438)scale(0.1 -0.1)\">\n","       <defs>\n","        <path id=\"DejaVuSans-30\" d=\"M 2034 4250 \n","Q 1547 4250 1301 3770 \n","Q 1056 3291 1056 2328 \n","Q 1056 1369 1301 889 \n","Q 1547 409 2034 409 \n","Q 2525 409 2770 889 \n","Q 3016 1369 3016 2328 \n","Q 3016 3291 2770 3770 \n","Q 2525 4250 2034 4250 \n","z\n","M 2034 4750 \n","Q 2819 4750 3233 4129 \n","Q 3647 3509 3647 2328 \n","Q 3647 1150 3233 529 \n","Q 2819 -91 2034 -91 \n","Q 1250 -91 836 529 \n","Q 422 1150 422 2328 \n","Q 422 3509 836 4129 \n","Q 1250 4750 2034 4750 \n","z\n","\" transform=\"scale(0.015625)\"/>\n","       </defs>\n","       <use xlink:href=\"#DejaVuSans-30\"/>\n","      </g>\n","     </g>\n","    </g>\n","    <g id=\"xtick_6\">\n","     <g id=\"line2d_11\">\n","      <path d=\"M 215.982798 143.1 \n","L 215.982798 7.2 \n","\" clip-path=\"url(#p8fd8832baa)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n","     </g>\n","     <g id=\"line2d_12\">\n","      <g>\n","       <use xlink:href=\"#m86a7e518fd\" x=\"215.982798\" y=\"143.1\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n","      </g>\n","     </g>\n","     <g id=\"text_6\">\n","      <!-- 2 -->\n","      <g transform=\"translate(212.801548 157.698438)scale(0.1 -0.1)\">\n","       <use xlink:href=\"#DejaVuSans-32\"/>\n","      </g>\n","     </g>\n","    </g>\n","    <g id=\"xtick_7\">\n","     <g id=\"line2d_13\">\n","      <path d=\"M 247.886743 143.1 \n","L 247.886743 7.2 \n","\" clip-path=\"url(#p8fd8832baa)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n","     </g>\n","     <g id=\"line2d_14\">\n","      <g>\n","       <use xlink:href=\"#m86a7e518fd\" x=\"247.886743\" y=\"143.1\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n","      </g>\n","     </g>\n","     <g id=\"text_7\">\n","      <!-- 4 -->\n","      <g transform=\"translate(244.705493 157.698438)scale(0.1 -0.1)\">\n","       <use xlink:href=\"#DejaVuSans-34\"/>\n","      </g>\n","     </g>\n","    </g>\n","    <g id=\"xtick_8\">\n","     <g id=\"line2d_15\">\n","      <path d=\"M 279.790689 143.1 \n","L 279.790689 7.2 \n","\" clip-path=\"url(#p8fd8832baa)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n","     </g>\n","     <g id=\"line2d_16\">\n","      <g>\n","       <use xlink:href=\"#m86a7e518fd\" x=\"279.790689\" y=\"143.1\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n","      </g>\n","     </g>\n","     <g id=\"text_8\">\n","      <!-- 6 -->\n","      <g transform=\"translate(276.609439 157.698438)scale(0.1 -0.1)\">\n","       <use xlink:href=\"#DejaVuSans-36\"/>\n","      </g>\n","     </g>\n","    </g>\n","    <g id=\"xtick_9\">\n","     <g id=\"line2d_17\">\n","      <path d=\"M 311.694635 143.1 \n","L 311.694635 7.2 \n","\" clip-path=\"url(#p8fd8832baa)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n","     </g>\n","     <g id=\"line2d_18\">\n","      <g>\n","       <use xlink:href=\"#m86a7e518fd\" x=\"311.694635\" y=\"143.1\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n","      </g>\n","     </g>\n","     <g id=\"text_9\">\n","      <!-- 8 -->\n","      <g transform=\"translate(308.513385 157.698438)scale(0.1 -0.1)\">\n","       <use xlink:href=\"#DejaVuSans-38\"/>\n","      </g>\n","     </g>\n","    </g>\n","    <g id=\"text_10\">\n","     <!-- x -->\n","     <g transform=\"translate(180.321875 171.376563)scale(0.1 -0.1)\">\n","      <defs>\n","       <path id=\"DejaVuSans-78\" d=\"M 3513 3500 \n","L 2247 1797 \n","L 3578 0 \n","L 2900 0 \n","L 1881 1375 \n","L 863 0 \n","L 184 0 \n","L 1544 1831 \n","L 300 3500 \n","L 978 3500 \n","L 1906 2253 \n","L 2834 3500 \n","L 3513 3500 \n","z\n","\" transform=\"scale(0.015625)\"/>\n","      </defs>\n","      <use xlink:href=\"#DejaVuSans-78\"/>\n","     </g>\n","    </g>\n","   </g>\n","   <g id=\"matplotlib.axis_2\">\n","    <g id=\"ytick_1\">\n","     <g id=\"line2d_19\">\n","      <path d=\"M 43.78125 136.922727 \n","L 322.78125 136.922727 \n","\" clip-path=\"url(#p8fd8832baa)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n","     </g>\n","     <g id=\"line2d_20\">\n","      <defs>\n","       <path id=\"m6c4f5831ff\" d=\"M 0 0 \n","L -3.5 0 \n","\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n","      </defs>\n","      <g>\n","       <use xlink:href=\"#m6c4f5831ff\" x=\"43.78125\" y=\"136.922727\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n","      </g>\n","     </g>\n","     <g id=\"text_11\">\n","      <!-- 0.0 -->\n","      <g transform=\"translate(20.878125 140.721946)scale(0.1 -0.1)\">\n","       <defs>\n","        <path id=\"DejaVuSans-2e\" d=\"M 684 794 \n","L 1344 794 \n","L 1344 0 \n","L 684 0 \n","L 684 794 \n","z\n","\" transform=\"scale(0.015625)\"/>\n","       </defs>\n","       <use xlink:href=\"#DejaVuSans-30\"/>\n","       <use xlink:href=\"#DejaVuSans-2e\" x=\"63.623047\"/>\n","       <use xlink:href=\"#DejaVuSans-30\" x=\"95.410156\"/>\n","      </g>\n","     </g>\n","    </g>\n","    <g id=\"ytick_2\">\n","     <g id=\"line2d_21\">\n","      <path d=\"M 43.78125 112.213636 \n","L 322.78125 112.213636 \n","\" clip-path=\"url(#p8fd8832baa)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n","     </g>\n","     <g id=\"line2d_22\">\n","      <g>\n","       <use xlink:href=\"#m6c4f5831ff\" x=\"43.78125\" y=\"112.213636\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n","      </g>\n","     </g>\n","     <g id=\"text_12\">\n","      <!-- 0.2 -->\n","      <g transform=\"translate(20.878125 116.012855)scale(0.1 -0.1)\">\n","       <use xlink:href=\"#DejaVuSans-30\"/>\n","       <use xlink:href=\"#DejaVuSans-2e\" x=\"63.623047\"/>\n","       <use xlink:href=\"#DejaVuSans-32\" x=\"95.410156\"/>\n","      </g>\n","     </g>\n","    </g>\n","    <g id=\"ytick_3\">\n","     <g id=\"line2d_23\">\n","      <path d=\"M 43.78125 87.504545 \n","L 322.78125 87.504545 \n","\" clip-path=\"url(#p8fd8832baa)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n","     </g>\n","     <g id=\"line2d_24\">\n","      <g>\n","       <use xlink:href=\"#m6c4f5831ff\" x=\"43.78125\" y=\"87.504545\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n","      </g>\n","     </g>\n","     <g id=\"text_13\">\n","      <!-- 0.4 -->\n","      <g transform=\"translate(20.878125 91.303764)scale(0.1 -0.1)\">\n","       <use xlink:href=\"#DejaVuSans-30\"/>\n","       <use xlink:href=\"#DejaVuSans-2e\" x=\"63.623047\"/>\n","       <use xlink:href=\"#DejaVuSans-34\" x=\"95.410156\"/>\n","      </g>\n","     </g>\n","    </g>\n","    <g id=\"ytick_4\">\n","     <g id=\"line2d_25\">\n","      <path d=\"M 43.78125 62.795455 \n","L 322.78125 62.795455 \n","\" clip-path=\"url(#p8fd8832baa)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n","     </g>\n","     <g id=\"line2d_26\">\n","      <g>\n","       <use xlink:href=\"#m6c4f5831ff\" x=\"43.78125\" y=\"62.795455\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n","      </g>\n","     </g>\n","     <g id=\"text_14\">\n","      <!-- 0.6 -->\n","      <g transform=\"translate(20.878125 66.594673)scale(0.1 -0.1)\">\n","       <use xlink:href=\"#DejaVuSans-30\"/>\n","       <use xlink:href=\"#DejaVuSans-2e\" x=\"63.623047\"/>\n","       <use xlink:href=\"#DejaVuSans-36\" x=\"95.410156\"/>\n","      </g>\n","     </g>\n","    </g>\n","    <g id=\"ytick_5\">\n","     <g id=\"line2d_27\">\n","      <path d=\"M 43.78125 38.086364 \n","L 322.78125 38.086364 \n","\" clip-path=\"url(#p8fd8832baa)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n","     </g>\n","     <g id=\"line2d_28\">\n","      <g>\n","       <use xlink:href=\"#m6c4f5831ff\" x=\"43.78125\" y=\"38.086364\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n","      </g>\n","     </g>\n","     <g id=\"text_15\">\n","      <!-- 0.8 -->\n","      <g transform=\"translate(20.878125 41.885582)scale(0.1 -0.1)\">\n","       <use xlink:href=\"#DejaVuSans-30\"/>\n","       <use xlink:href=\"#DejaVuSans-2e\" x=\"63.623047\"/>\n","       <use xlink:href=\"#DejaVuSans-38\" x=\"95.410156\"/>\n","      </g>\n","     </g>\n","    </g>\n","    <g id=\"ytick_6\">\n","     <g id=\"line2d_29\">\n","      <path d=\"M 43.78125 13.377273 \n","L 322.78125 13.377273 \n","\" clip-path=\"url(#p8fd8832baa)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n","     </g>\n","     <g id=\"line2d_30\">\n","      <g>\n","       <use xlink:href=\"#m6c4f5831ff\" x=\"43.78125\" y=\"13.377273\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n","      </g>\n","     </g>\n","     <g id=\"text_16\">\n","      <!-- 1.0 -->\n","      <g transform=\"translate(20.878125 17.176491)scale(0.1 -0.1)\">\n","       <defs>\n","        <path id=\"DejaVuSans-31\" d=\"M 794 531 \n","L 1825 531 \n","L 1825 4091 \n","L 703 3866 \n","L 703 4441 \n","L 1819 4666 \n","L 2450 4666 \n","L 2450 531 \n","L 3481 531 \n","L 3481 0 \n","L 794 0 \n","L 794 531 \n","z\n","\" transform=\"scale(0.015625)\"/>\n","       </defs>\n","       <use xlink:href=\"#DejaVuSans-31\"/>\n","       <use xlink:href=\"#DejaVuSans-2e\" x=\"63.623047\"/>\n","       <use xlink:href=\"#DejaVuSans-30\" x=\"95.410156\"/>\n","      </g>\n","     </g>\n","    </g>\n","    <g id=\"text_17\">\n","     <!-- grad of relu -->\n","     <g transform=\"translate(14.798438 104.192969)rotate(-90)scale(0.1 -0.1)\">\n","      <defs>\n","       <path id=\"DejaVuSans-67\" d=\"M 2906 1791 \n","Q 2906 2416 2648 2759 \n","Q 2391 3103 1925 3103 \n","Q 1463 3103 1205 2759 \n","Q 947 2416 947 1791 \n","Q 947 1169 1205 825 \n","Q 1463 481 1925 481 \n","Q 2391 481 2648 825 \n","Q 2906 1169 2906 1791 \n","z\n","M 3481 434 \n","Q 3481 -459 3084 -895 \n","Q 2688 -1331 1869 -1331 \n","Q 1566 -1331 1297 -1286 \n","Q 1028 -1241 775 -1147 \n","L 775 -588 \n","Q 1028 -725 1275 -790 \n","Q 1522 -856 1778 -856 \n","Q 2344 -856 2625 -561 \n","Q 2906 -266 2906 331 \n","L 2906 616 \n","Q 2728 306 2450 153 \n","Q 2172 0 1784 0 \n","Q 1141 0 747 490 \n","Q 353 981 353 1791 \n","Q 353 2603 747 3093 \n","Q 1141 3584 1784 3584 \n","Q 2172 3584 2450 3431 \n","Q 2728 3278 2906 2969 \n","L 2906 3500 \n","L 3481 3500 \n","L 3481 434 \n","z\n","\" transform=\"scale(0.015625)\"/>\n","       <path id=\"DejaVuSans-72\" d=\"M 2631 2963 \n","Q 2534 3019 2420 3045 \n","Q 2306 3072 2169 3072 \n","Q 1681 3072 1420 2755 \n","Q 1159 2438 1159 1844 \n","L 1159 0 \n","L 581 0 \n","L 581 3500 \n","L 1159 3500 \n","L 1159 2956 \n","Q 1341 3275 1631 3429 \n","Q 1922 3584 2338 3584 \n","Q 2397 3584 2469 3576 \n","Q 2541 3569 2628 3553 \n","L 2631 2963 \n","z\n","\" transform=\"scale(0.015625)\"/>\n","       <path id=\"DejaVuSans-61\" d=\"M 2194 1759 \n","Q 1497 1759 1228 1600 \n","Q 959 1441 959 1056 \n","Q 959 750 1161 570 \n","Q 1363 391 1709 391 \n","Q 2188 391 2477 730 \n","Q 2766 1069 2766 1631 \n","L 2766 1759 \n","L 2194 1759 \n","z\n","M 3341 1997 \n","L 3341 0 \n","L 2766 0 \n","L 2766 531 \n","Q 2569 213 2275 61 \n","Q 1981 -91 1556 -91 \n","Q 1019 -91 701 211 \n","Q 384 513 384 1019 \n","Q 384 1609 779 1909 \n","Q 1175 2209 1959 2209 \n","L 2766 2209 \n","L 2766 2266 \n","Q 2766 2663 2505 2880 \n","Q 2244 3097 1772 3097 \n","Q 1472 3097 1187 3025 \n","Q 903 2953 641 2809 \n","L 641 3341 \n","Q 956 3463 1253 3523 \n","Q 1550 3584 1831 3584 \n","Q 2591 3584 2966 3190 \n","Q 3341 2797 3341 1997 \n","z\n","\" transform=\"scale(0.015625)\"/>\n","       <path id=\"DejaVuSans-64\" d=\"M 2906 2969 \n","L 2906 4863 \n","L 3481 4863 \n","L 3481 0 \n","L 2906 0 \n","L 2906 525 \n","Q 2725 213 2448 61 \n","Q 2172 -91 1784 -91 \n","Q 1150 -91 751 415 \n","Q 353 922 353 1747 \n","Q 353 2572 751 3078 \n","Q 1150 3584 1784 3584 \n","Q 2172 3584 2448 3432 \n","Q 2725 3281 2906 2969 \n","z\n","M 947 1747 \n","Q 947 1113 1208 752 \n","Q 1469 391 1925 391 \n","Q 2381 391 2643 752 \n","Q 2906 1113 2906 1747 \n","Q 2906 2381 2643 2742 \n","Q 2381 3103 1925 3103 \n","Q 1469 3103 1208 2742 \n","Q 947 2381 947 1747 \n","z\n","\" transform=\"scale(0.015625)\"/>\n","       <path id=\"DejaVuSans-20\" transform=\"scale(0.015625)\"/>\n","       <path id=\"DejaVuSans-6f\" d=\"M 1959 3097 \n","Q 1497 3097 1228 2736 \n","Q 959 2375 959 1747 \n","Q 959 1119 1226 758 \n","Q 1494 397 1959 397 \n","Q 2419 397 2687 759 \n","Q 2956 1122 2956 1747 \n","Q 2956 2369 2687 2733 \n","Q 2419 3097 1959 3097 \n","z\n","M 1959 3584 \n","Q 2709 3584 3137 3096 \n","Q 3566 2609 3566 1747 \n","Q 3566 888 3137 398 \n","Q 2709 -91 1959 -91 \n","Q 1206 -91 779 398 \n","Q 353 888 353 1747 \n","Q 353 2609 779 3096 \n","Q 1206 3584 1959 3584 \n","z\n","\" transform=\"scale(0.015625)\"/>\n","       <path id=\"DejaVuSans-66\" d=\"M 2375 4863 \n","L 2375 4384 \n","L 1825 4384 \n","Q 1516 4384 1395 4259 \n","Q 1275 4134 1275 3809 \n","L 1275 3500 \n","L 2222 3500 \n","L 2222 3053 \n","L 1275 3053 \n","L 1275 0 \n","L 697 0 \n","L 697 3053 \n","L 147 3053 \n","L 147 3500 \n","L 697 3500 \n","L 697 3744 \n","Q 697 4328 969 4595 \n","Q 1241 4863 1831 4863 \n","L 2375 4863 \n","z\n","\" transform=\"scale(0.015625)\"/>\n","       <path id=\"DejaVuSans-65\" d=\"M 3597 1894 \n","L 3597 1613 \n","L 953 1613 \n","Q 991 1019 1311 708 \n","Q 1631 397 2203 397 \n","Q 2534 397 2845 478 \n","Q 3156 559 3463 722 \n","L 3463 178 \n","Q 3153 47 2828 -22 \n","Q 2503 -91 2169 -91 \n","Q 1331 -91 842 396 \n","Q 353 884 353 1716 \n","Q 353 2575 817 3079 \n","Q 1281 3584 2069 3584 \n","Q 2775 3584 3186 3129 \n","Q 3597 2675 3597 1894 \n","z\n","M 3022 2063 \n","Q 3016 2534 2758 2815 \n","Q 2500 3097 2075 3097 \n","Q 1594 3097 1305 2825 \n","Q 1016 2553 972 2059 \n","L 3022 2063 \n","z\n","\" transform=\"scale(0.015625)\"/>\n","       <path id=\"DejaVuSans-6c\" d=\"M 603 4863 \n","L 1178 4863 \n","L 1178 0 \n","L 603 0 \n","L 603 4863 \n","z\n","\" transform=\"scale(0.015625)\"/>\n","       <path id=\"DejaVuSans-75\" d=\"M 544 1381 \n","L 544 3500 \n","L 1119 3500 \n","L 1119 1403 \n","Q 1119 906 1312 657 \n","Q 1506 409 1894 409 \n","Q 2359 409 2629 706 \n","Q 2900 1003 2900 1516 \n","L 2900 3500 \n","L 3475 3500 \n","L 3475 0 \n","L 2900 0 \n","L 2900 538 \n","Q 2691 219 2414 64 \n","Q 2138 -91 1772 -91 \n","Q 1169 -91 856 284 \n","Q 544 659 544 1381 \n","z\n","M 1991 3584 \n","L 1991 3584 \n","z\n","\" transform=\"scale(0.015625)\"/>\n","      </defs>\n","      <use xlink:href=\"#DejaVuSans-67\"/>\n","      <use xlink:href=\"#DejaVuSans-72\" x=\"63.476562\"/>\n","      <use xlink:href=\"#DejaVuSans-61\" x=\"104.589844\"/>\n","      <use xlink:href=\"#DejaVuSans-64\" x=\"165.869141\"/>\n","      <use xlink:href=\"#DejaVuSans-20\" x=\"229.345703\"/>\n","      <use xlink:href=\"#DejaVuSans-6f\" x=\"261.132812\"/>\n","      <use xlink:href=\"#DejaVuSans-66\" x=\"322.314453\"/>\n","      <use xlink:href=\"#DejaVuSans-20\" x=\"357.519531\"/>\n","      <use xlink:href=\"#DejaVuSans-72\" x=\"389.306641\"/>\n","      <use xlink:href=\"#DejaVuSans-65\" x=\"428.169922\"/>\n","      <use xlink:href=\"#DejaVuSans-6c\" x=\"489.693359\"/>\n","      <use xlink:href=\"#DejaVuSans-75\" x=\"517.476562\"/>\n","     </g>\n","    </g>\n","   </g>\n","   <g id=\"line2d_31\">\n","    <path d=\"M 56.463068 136.922727 \n","L 182.483654 136.922727 \n","L 184.078852 13.377273 \n","L 310.099432 13.377273 \n","L 310.099432 13.377273 \n","\" clip-path=\"url(#p8fd8832baa)\" style=\"fill: none; stroke: #1f77b4; stroke-width: 1.5; stroke-linecap: square\"/>\n","   </g>\n","   <g id=\"patch_3\">\n","    <path d=\"M 43.78125 143.1 \n","L 43.78125 7.2 \n","\" style=\"fill: none; stroke: #000000; stroke-width: 0.8; stroke-linejoin: miter; stroke-linecap: square\"/>\n","   </g>\n","   <g id=\"patch_4\">\n","    <path d=\"M 322.78125 143.1 \n","L 322.78125 7.2 \n","\" style=\"fill: none; stroke: #000000; stroke-width: 0.8; stroke-linejoin: miter; stroke-linecap: square\"/>\n","   </g>\n","   <g id=\"patch_5\">\n","    <path d=\"M 43.78125 143.1 \n","L 322.78125 143.1 \n","\" style=\"fill: none; stroke: #000000; stroke-width: 0.8; stroke-linejoin: miter; stroke-linecap: square\"/>\n","   </g>\n","   <g id=\"patch_6\">\n","    <path d=\"M 43.78125 7.2 \n","L 322.78125 7.2 \n","\" style=\"fill: none; stroke: #000000; stroke-width: 0.8; stroke-linejoin: miter; stroke-linecap: square\"/>\n","   </g>\n","  </g>\n"," </g>\n"," <defs>\n","  <clipPath id=\"p8fd8832baa\">\n","   <rect x=\"43.78125\" y=\"7.2\" width=\"279\" height=\"135.9\"/>\n","  </clipPath>\n"," </defs>\n","</svg>\n"],"text/plain":["<Figure size 360x180 with 1 Axes>"]},"metadata":{"needs_background":"light"},"output_type":"display_data"}],"source":["y.backward(torch.ones_like(x), retain_graph=True)\n","d2l.plot(x.detach(), x.grad, 'x', 'grad of relu', figsize=(5, 2.5))"]},{"cell_type":"markdown","id":"167657ef","metadata":{"origin_pos":12,"id":"167657ef"},"source":["使用ReLU的原因是，它求导表现得特别好：要么让参数消失，要么让参数通过。\n","这使得优化表现得更好，并且ReLU减轻了困扰以往神经网络的梯度消失问题（稍后将详细介绍）。\n","\n","注意，ReLU函数有许多变体，包括*参数化ReLU*（Parameterized ReLU，*pReLU*）\n","函数。\n","该变体为ReLU添加了一个线性项，因此即使参数是负的，某些信息仍然可以通过：\n","\n","$$\\operatorname{pReLU}(x) = \\max(0, x) + \\alpha \\min(0, x).$$\n","\n","### sigmoid函数\n","\n","[**对于一个定义域在$\\mathbb{R}$中的输入，\n","*sigmoid函数*将输入变换为区间(0, 1)上的输出**]。\n","因此，sigmoid通常称为*挤压函数*（squashing function）：\n","它将范围（-inf, inf）中的任意输入压缩到区间（0, 1）中的某个值：\n","\n","**$$\\operatorname{sigmoid}(x) = \\frac{1}{1 + \\exp(-x)}.$$**\n","\n","在最早的神经网络中，科学家们感兴趣的是对“激发”或“不激发”的生物神经元进行建模。\n","因此，这一领域的先驱可以一直追溯到人工神经元的发明者麦卡洛克和皮茨，他们专注于阈值单元。\n","阈值单元在其输入低于某个阈值时取值0，当输入超过阈值时取值1。\n","\n","当人们逐渐关注到到基于梯度的学习时，\n","sigmoid函数是一个自然的选择，因为它是一个平滑的、可微的阈值单元近似。\n","当我们想要将输出视作二元分类问题的概率时，\n","sigmoid仍然被广泛用作输出单元上的激活函数\n","（你可以将sigmoid视为softmax的特例）。\n","然而，sigmoid在隐藏层中已经较少使用，\n","它在大部分时候被更简单、更容易训练的ReLU所取代。\n","在后面关于循环神经网络的章节中，我们将描述利用sigmoid单元来控制时序信息流的架构。\n","\n","下面，我们绘制sigmoid函数。\n","注意，当输入接近0时，sigmoid函数接近线性变换。\n"]},{"cell_type":"code","execution_count":null,"id":"059a6735","metadata":{"execution":{"iopub.execute_input":"2022-07-31T22:40:54.381679Z","iopub.status.busy":"2022-07-31T22:40:54.381185Z","iopub.status.idle":"2022-07-31T22:40:54.581278Z","shell.execute_reply":"2022-07-31T22:40:54.580607Z"},"origin_pos":14,"tab":["pytorch"],"id":"059a6735","outputId":"cb72562c-2a7b-4785-8f0b-0996e9e16399"},"outputs":[{"data":{"image/svg+xml":["<?xml version=\"1.0\" encoding=\"utf-8\" standalone=\"no\"?>\n","<!DOCTYPE svg PUBLIC \"-//W3C//DTD SVG 1.1//EN\"\n","  \"http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd\">\n","<svg xmlns:xlink=\"http://www.w3.org/1999/xlink\" width=\"329.98125pt\" height=\"180.65625pt\" viewBox=\"0 0 329.98125 180.65625\" xmlns=\"http://www.w3.org/2000/svg\" version=\"1.1\">\n"," <metadata>\n","  <rdf:RDF xmlns:dc=\"http://purl.org/dc/elements/1.1/\" xmlns:cc=\"http://creativecommons.org/ns#\" xmlns:rdf=\"http://www.w3.org/1999/02/22-rdf-syntax-ns#\">\n","   <cc:Work>\n","    <dc:type rdf:resource=\"http://purl.org/dc/dcmitype/StillImage\"/>\n","    <dc:date>2022-07-31T22:40:54.537678</dc:date>\n","    <dc:format>image/svg+xml</dc:format>\n","    <dc:creator>\n","     <cc:Agent>\n","      <dc:title>Matplotlib v3.5.1, https://matplotlib.org/</dc:title>\n","     </cc:Agent>\n","    </dc:creator>\n","   </cc:Work>\n","  </rdf:RDF>\n"," </metadata>\n"," <defs>\n","  <style type=\"text/css\">*{stroke-linejoin: round; stroke-linecap: butt}</style>\n"," </defs>\n"," <g id=\"figure_1\">\n","  <g id=\"patch_1\">\n","   <path d=\"M 0 180.65625 \n","L 329.98125 180.65625 \n","L 329.98125 0 \n","L 0 0 \n","L 0 180.65625 \n","z\n","\" style=\"fill: none\"/>\n","  </g>\n","  <g id=\"axes_1\">\n","   <g id=\"patch_2\">\n","    <path d=\"M 43.78125 143.1 \n","L 322.78125 143.1 \n","L 322.78125 7.2 \n","L 43.78125 7.2 \n","z\n","\" style=\"fill: #ffffff\"/>\n","   </g>\n","   <g id=\"matplotlib.axis_1\">\n","    <g id=\"xtick_1\">\n","     <g id=\"line2d_1\">\n","      <path d=\"M 56.463068 143.1 \n","L 56.463068 7.2 \n","\" clip-path=\"url(#p178e410158)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n","     </g>\n","     <g id=\"line2d_2\">\n","      <defs>\n","       <path id=\"m5fef8c8f1c\" d=\"M 0 0 \n","L 0 3.5 \n","\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n","      </defs>\n","      <g>\n","       <use xlink:href=\"#m5fef8c8f1c\" x=\"56.463068\" y=\"143.1\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n","      </g>\n","     </g>\n","     <g id=\"text_1\">\n","      <!-- −8 -->\n","      <g transform=\"translate(49.091974 157.698438)scale(0.1 -0.1)\">\n","       <defs>\n","        <path id=\"DejaVuSans-2212\" d=\"M 678 2272 \n","L 4684 2272 \n","L 4684 1741 \n","L 678 1741 \n","L 678 2272 \n","z\n","\" transform=\"scale(0.015625)\"/>\n","        <path id=\"DejaVuSans-38\" d=\"M 2034 2216 \n","Q 1584 2216 1326 1975 \n","Q 1069 1734 1069 1313 \n","Q 1069 891 1326 650 \n","Q 1584 409 2034 409 \n","Q 2484 409 2743 651 \n","Q 3003 894 3003 1313 \n","Q 3003 1734 2745 1975 \n","Q 2488 2216 2034 2216 \n","z\n","M 1403 2484 \n","Q 997 2584 770 2862 \n","Q 544 3141 544 3541 \n","Q 544 4100 942 4425 \n","Q 1341 4750 2034 4750 \n","Q 2731 4750 3128 4425 \n","Q 3525 4100 3525 3541 \n","Q 3525 3141 3298 2862 \n","Q 3072 2584 2669 2484 \n","Q 3125 2378 3379 2068 \n","Q 3634 1759 3634 1313 \n","Q 3634 634 3220 271 \n","Q 2806 -91 2034 -91 \n","Q 1263 -91 848 271 \n","Q 434 634 434 1313 \n","Q 434 1759 690 2068 \n","Q 947 2378 1403 2484 \n","z\n","M 1172 3481 \n","Q 1172 3119 1398 2916 \n","Q 1625 2713 2034 2713 \n","Q 2441 2713 2670 2916 \n","Q 2900 3119 2900 3481 \n","Q 2900 3844 2670 4047 \n","Q 2441 4250 2034 4250 \n","Q 1625 4250 1398 4047 \n","Q 1172 3844 1172 3481 \n","z\n","\" transform=\"scale(0.015625)\"/>\n","       </defs>\n","       <use xlink:href=\"#DejaVuSans-2212\"/>\n","       <use xlink:href=\"#DejaVuSans-38\" x=\"83.789062\"/>\n","      </g>\n","     </g>\n","    </g>\n","    <g id=\"xtick_2\">\n","     <g id=\"line2d_3\">\n","      <path d=\"M 88.367014 143.1 \n","L 88.367014 7.2 \n","\" clip-path=\"url(#p178e410158)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n","     </g>\n","     <g id=\"line2d_4\">\n","      <g>\n","       <use xlink:href=\"#m5fef8c8f1c\" x=\"88.367014\" y=\"143.1\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n","      </g>\n","     </g>\n","     <g id=\"text_2\">\n","      <!-- −6 -->\n","      <g transform=\"translate(80.99592 157.698438)scale(0.1 -0.1)\">\n","       <defs>\n","        <path id=\"DejaVuSans-36\" d=\"M 2113 2584 \n","Q 1688 2584 1439 2293 \n","Q 1191 2003 1191 1497 \n","Q 1191 994 1439 701 \n","Q 1688 409 2113 409 \n","Q 2538 409 2786 701 \n","Q 3034 994 3034 1497 \n","Q 3034 2003 2786 2293 \n","Q 2538 2584 2113 2584 \n","z\n","M 3366 4563 \n","L 3366 3988 \n","Q 3128 4100 2886 4159 \n","Q 2644 4219 2406 4219 \n","Q 1781 4219 1451 3797 \n","Q 1122 3375 1075 2522 \n","Q 1259 2794 1537 2939 \n","Q 1816 3084 2150 3084 \n","Q 2853 3084 3261 2657 \n","Q 3669 2231 3669 1497 \n","Q 3669 778 3244 343 \n","Q 2819 -91 2113 -91 \n","Q 1303 -91 875 529 \n","Q 447 1150 447 2328 \n","Q 447 3434 972 4092 \n","Q 1497 4750 2381 4750 \n","Q 2619 4750 2861 4703 \n","Q 3103 4656 3366 4563 \n","z\n","\" transform=\"scale(0.015625)\"/>\n","       </defs>\n","       <use xlink:href=\"#DejaVuSans-2212\"/>\n","       <use xlink:href=\"#DejaVuSans-36\" x=\"83.789062\"/>\n","      </g>\n","     </g>\n","    </g>\n","    <g id=\"xtick_3\">\n","     <g id=\"line2d_5\">\n","      <path d=\"M 120.27096 143.1 \n","L 120.27096 7.2 \n","\" clip-path=\"url(#p178e410158)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n","     </g>\n","     <g id=\"line2d_6\">\n","      <g>\n","       <use xlink:href=\"#m5fef8c8f1c\" x=\"120.27096\" y=\"143.1\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n","      </g>\n","     </g>\n","     <g id=\"text_3\">\n","      <!-- −4 -->\n","      <g transform=\"translate(112.899866 157.698438)scale(0.1 -0.1)\">\n","       <defs>\n","        <path id=\"DejaVuSans-34\" d=\"M 2419 4116 \n","L 825 1625 \n","L 2419 1625 \n","L 2419 4116 \n","z\n","M 2253 4666 \n","L 3047 4666 \n","L 3047 1625 \n","L 3713 1625 \n","L 3713 1100 \n","L 3047 1100 \n","L 3047 0 \n","L 2419 0 \n","L 2419 1100 \n","L 313 1100 \n","L 313 1709 \n","L 2253 4666 \n","z\n","\" transform=\"scale(0.015625)\"/>\n","       </defs>\n","       <use xlink:href=\"#DejaVuSans-2212\"/>\n","       <use xlink:href=\"#DejaVuSans-34\" x=\"83.789062\"/>\n","      </g>\n","     </g>\n","    </g>\n","    <g id=\"xtick_4\">\n","     <g id=\"line2d_7\">\n","      <path d=\"M 152.174906 143.1 \n","L 152.174906 7.2 \n","\" clip-path=\"url(#p178e410158)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n","     </g>\n","     <g id=\"line2d_8\">\n","      <g>\n","       <use xlink:href=\"#m5fef8c8f1c\" x=\"152.174906\" y=\"143.1\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n","      </g>\n","     </g>\n","     <g id=\"text_4\">\n","      <!-- −2 -->\n","      <g transform=\"translate(144.803812 157.698438)scale(0.1 -0.1)\">\n","       <defs>\n","        <path id=\"DejaVuSans-32\" d=\"M 1228 531 \n","L 3431 531 \n","L 3431 0 \n","L 469 0 \n","L 469 531 \n","Q 828 903 1448 1529 \n","Q 2069 2156 2228 2338 \n","Q 2531 2678 2651 2914 \n","Q 2772 3150 2772 3378 \n","Q 2772 3750 2511 3984 \n","Q 2250 4219 1831 4219 \n","Q 1534 4219 1204 4116 \n","Q 875 4013 500 3803 \n","L 500 4441 \n","Q 881 4594 1212 4672 \n","Q 1544 4750 1819 4750 \n","Q 2544 4750 2975 4387 \n","Q 3406 4025 3406 3419 \n","Q 3406 3131 3298 2873 \n","Q 3191 2616 2906 2266 \n","Q 2828 2175 2409 1742 \n","Q 1991 1309 1228 531 \n","z\n","\" transform=\"scale(0.015625)\"/>\n","       </defs>\n","       <use xlink:href=\"#DejaVuSans-2212\"/>\n","       <use xlink:href=\"#DejaVuSans-32\" x=\"83.789062\"/>\n","      </g>\n","     </g>\n","    </g>\n","    <g id=\"xtick_5\">\n","     <g id=\"line2d_9\">\n","      <path d=\"M 184.078852 143.1 \n","L 184.078852 7.2 \n","\" clip-path=\"url(#p178e410158)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n","     </g>\n","     <g id=\"line2d_10\">\n","      <g>\n","       <use xlink:href=\"#m5fef8c8f1c\" x=\"184.078852\" y=\"143.1\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n","      </g>\n","     </g>\n","     <g id=\"text_5\">\n","      <!-- 0 -->\n","      <g transform=\"translate(180.897602 157.698438)scale(0.1 -0.1)\">\n","       <defs>\n","        <path id=\"DejaVuSans-30\" d=\"M 2034 4250 \n","Q 1547 4250 1301 3770 \n","Q 1056 3291 1056 2328 \n","Q 1056 1369 1301 889 \n","Q 1547 409 2034 409 \n","Q 2525 409 2770 889 \n","Q 3016 1369 3016 2328 \n","Q 3016 3291 2770 3770 \n","Q 2525 4250 2034 4250 \n","z\n","M 2034 4750 \n","Q 2819 4750 3233 4129 \n","Q 3647 3509 3647 2328 \n","Q 3647 1150 3233 529 \n","Q 2819 -91 2034 -91 \n","Q 1250 -91 836 529 \n","Q 422 1150 422 2328 \n","Q 422 3509 836 4129 \n","Q 1250 4750 2034 4750 \n","z\n","\" transform=\"scale(0.015625)\"/>\n","       </defs>\n","       <use xlink:href=\"#DejaVuSans-30\"/>\n","      </g>\n","     </g>\n","    </g>\n","    <g id=\"xtick_6\">\n","     <g id=\"line2d_11\">\n","      <path d=\"M 215.982798 143.1 \n","L 215.982798 7.2 \n","\" clip-path=\"url(#p178e410158)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n","     </g>\n","     <g id=\"line2d_12\">\n","      <g>\n","       <use xlink:href=\"#m5fef8c8f1c\" x=\"215.982798\" y=\"143.1\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n","      </g>\n","     </g>\n","     <g id=\"text_6\">\n","      <!-- 2 -->\n","      <g transform=\"translate(212.801548 157.698438)scale(0.1 -0.1)\">\n","       <use xlink:href=\"#DejaVuSans-32\"/>\n","      </g>\n","     </g>\n","    </g>\n","    <g id=\"xtick_7\">\n","     <g id=\"line2d_13\">\n","      <path d=\"M 247.886743 143.1 \n","L 247.886743 7.2 \n","\" clip-path=\"url(#p178e410158)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n","     </g>\n","     <g id=\"line2d_14\">\n","      <g>\n","       <use xlink:href=\"#m5fef8c8f1c\" x=\"247.886743\" y=\"143.1\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n","      </g>\n","     </g>\n","     <g id=\"text_7\">\n","      <!-- 4 -->\n","      <g transform=\"translate(244.705493 157.698438)scale(0.1 -0.1)\">\n","       <use xlink:href=\"#DejaVuSans-34\"/>\n","      </g>\n","     </g>\n","    </g>\n","    <g id=\"xtick_8\">\n","     <g id=\"line2d_15\">\n","      <path d=\"M 279.790689 143.1 \n","L 279.790689 7.2 \n","\" clip-path=\"url(#p178e410158)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n","     </g>\n","     <g id=\"line2d_16\">\n","      <g>\n","       <use xlink:href=\"#m5fef8c8f1c\" x=\"279.790689\" y=\"143.1\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n","      </g>\n","     </g>\n","     <g id=\"text_8\">\n","      <!-- 6 -->\n","      <g transform=\"translate(276.609439 157.698438)scale(0.1 -0.1)\">\n","       <use xlink:href=\"#DejaVuSans-36\"/>\n","      </g>\n","     </g>\n","    </g>\n","    <g id=\"xtick_9\">\n","     <g id=\"line2d_17\">\n","      <path d=\"M 311.694635 143.1 \n","L 311.694635 7.2 \n","\" clip-path=\"url(#p178e410158)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n","     </g>\n","     <g id=\"line2d_18\">\n","      <g>\n","       <use xlink:href=\"#m5fef8c8f1c\" x=\"311.694635\" y=\"143.1\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n","      </g>\n","     </g>\n","     <g id=\"text_9\">\n","      <!-- 8 -->\n","      <g transform=\"translate(308.513385 157.698438)scale(0.1 -0.1)\">\n","       <use xlink:href=\"#DejaVuSans-38\"/>\n","      </g>\n","     </g>\n","    </g>\n","    <g id=\"text_10\">\n","     <!-- x -->\n","     <g transform=\"translate(180.321875 171.376563)scale(0.1 -0.1)\">\n","      <defs>\n","       <path id=\"DejaVuSans-78\" d=\"M 3513 3500 \n","L 2247 1797 \n","L 3578 0 \n","L 2900 0 \n","L 1881 1375 \n","L 863 0 \n","L 184 0 \n","L 1544 1831 \n","L 300 3500 \n","L 978 3500 \n","L 1906 2253 \n","L 2834 3500 \n","L 3513 3500 \n","z\n","\" transform=\"scale(0.015625)\"/>\n","      </defs>\n","      <use xlink:href=\"#DejaVuSans-78\"/>\n","     </g>\n","    </g>\n","   </g>\n","   <g id=\"matplotlib.axis_2\">\n","    <g id=\"ytick_1\">\n","     <g id=\"line2d_19\">\n","      <path d=\"M 43.78125 136.964188 \n","L 322.78125 136.964188 \n","\" clip-path=\"url(#p178e410158)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n","     </g>\n","     <g id=\"line2d_20\">\n","      <defs>\n","       <path id=\"mdc73d64f2a\" d=\"M 0 0 \n","L -3.5 0 \n","\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n","      </defs>\n","      <g>\n","       <use xlink:href=\"#mdc73d64f2a\" x=\"43.78125\" y=\"136.964188\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n","      </g>\n","     </g>\n","     <g id=\"text_11\">\n","      <!-- 0.0 -->\n","      <g transform=\"translate(20.878125 140.763406)scale(0.1 -0.1)\">\n","       <defs>\n","        <path id=\"DejaVuSans-2e\" d=\"M 684 794 \n","L 1344 794 \n","L 1344 0 \n","L 684 0 \n","L 684 794 \n","z\n","\" transform=\"scale(0.015625)\"/>\n","       </defs>\n","       <use xlink:href=\"#DejaVuSans-30\"/>\n","       <use xlink:href=\"#DejaVuSans-2e\" x=\"63.623047\"/>\n","       <use xlink:href=\"#DejaVuSans-30\" x=\"95.410156\"/>\n","      </g>\n","     </g>\n","    </g>\n","    <g id=\"ytick_2\">\n","     <g id=\"line2d_21\">\n","      <path d=\"M 43.78125 112.23764 \n","L 322.78125 112.23764 \n","\" clip-path=\"url(#p178e410158)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n","     </g>\n","     <g id=\"line2d_22\">\n","      <g>\n","       <use xlink:href=\"#mdc73d64f2a\" x=\"43.78125\" y=\"112.23764\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n","      </g>\n","     </g>\n","     <g id=\"text_12\">\n","      <!-- 0.2 -->\n","      <g transform=\"translate(20.878125 116.036859)scale(0.1 -0.1)\">\n","       <use xlink:href=\"#DejaVuSans-30\"/>\n","       <use xlink:href=\"#DejaVuSans-2e\" x=\"63.623047\"/>\n","       <use xlink:href=\"#DejaVuSans-32\" x=\"95.410156\"/>\n","      </g>\n","     </g>\n","    </g>\n","    <g id=\"ytick_3\">\n","     <g id=\"line2d_23\">\n","      <path d=\"M 43.78125 87.511093 \n","L 322.78125 87.511093 \n","\" clip-path=\"url(#p178e410158)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n","     </g>\n","     <g id=\"line2d_24\">\n","      <g>\n","       <use xlink:href=\"#mdc73d64f2a\" x=\"43.78125\" y=\"87.511093\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n","      </g>\n","     </g>\n","     <g id=\"text_13\">\n","      <!-- 0.4 -->\n","      <g transform=\"translate(20.878125 91.310312)scale(0.1 -0.1)\">\n","       <use xlink:href=\"#DejaVuSans-30\"/>\n","       <use xlink:href=\"#DejaVuSans-2e\" x=\"63.623047\"/>\n","       <use xlink:href=\"#DejaVuSans-34\" x=\"95.410156\"/>\n","      </g>\n","     </g>\n","    </g>\n","    <g id=\"ytick_4\">\n","     <g id=\"line2d_25\">\n","      <path d=\"M 43.78125 62.784546 \n","L 322.78125 62.784546 \n","\" clip-path=\"url(#p178e410158)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n","     </g>\n","     <g id=\"line2d_26\">\n","      <g>\n","       <use xlink:href=\"#mdc73d64f2a\" x=\"43.78125\" y=\"62.784546\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n","      </g>\n","     </g>\n","     <g id=\"text_14\">\n","      <!-- 0.6 -->\n","      <g transform=\"translate(20.878125 66.583765)scale(0.1 -0.1)\">\n","       <use xlink:href=\"#DejaVuSans-30\"/>\n","       <use xlink:href=\"#DejaVuSans-2e\" x=\"63.623047\"/>\n","       <use xlink:href=\"#DejaVuSans-36\" x=\"95.410156\"/>\n","      </g>\n","     </g>\n","    </g>\n","    <g id=\"ytick_5\">\n","     <g id=\"line2d_27\">\n","      <path d=\"M 43.78125 38.057999 \n","L 322.78125 38.057999 \n","\" clip-path=\"url(#p178e410158)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n","     </g>\n","     <g id=\"line2d_28\">\n","      <g>\n","       <use xlink:href=\"#mdc73d64f2a\" x=\"43.78125\" y=\"38.057999\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n","      </g>\n","     </g>\n","     <g id=\"text_15\">\n","      <!-- 0.8 -->\n","      <g transform=\"translate(20.878125 41.857218)scale(0.1 -0.1)\">\n","       <use xlink:href=\"#DejaVuSans-30\"/>\n","       <use xlink:href=\"#DejaVuSans-2e\" x=\"63.623047\"/>\n","       <use xlink:href=\"#DejaVuSans-38\" x=\"95.410156\"/>\n","      </g>\n","     </g>\n","    </g>\n","    <g id=\"ytick_6\">\n","     <g id=\"line2d_29\">\n","      <path d=\"M 43.78125 13.331452 \n","L 322.78125 13.331452 \n","\" clip-path=\"url(#p178e410158)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n","     </g>\n","     <g id=\"line2d_30\">\n","      <g>\n","       <use xlink:href=\"#mdc73d64f2a\" x=\"43.78125\" y=\"13.331452\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n","      </g>\n","     </g>\n","     <g id=\"text_16\">\n","      <!-- 1.0 -->\n","      <g transform=\"translate(20.878125 17.130671)scale(0.1 -0.1)\">\n","       <defs>\n","        <path id=\"DejaVuSans-31\" d=\"M 794 531 \n","L 1825 531 \n","L 1825 4091 \n","L 703 3866 \n","L 703 4441 \n","L 1819 4666 \n","L 2450 4666 \n","L 2450 531 \n","L 3481 531 \n","L 3481 0 \n","L 794 0 \n","L 794 531 \n","z\n","\" transform=\"scale(0.015625)\"/>\n","       </defs>\n","       <use xlink:href=\"#DejaVuSans-31\"/>\n","       <use xlink:href=\"#DejaVuSans-2e\" x=\"63.623047\"/>\n","       <use xlink:href=\"#DejaVuSans-30\" x=\"95.410156\"/>\n","      </g>\n","     </g>\n","    </g>\n","    <g id=\"text_17\">\n","     <!-- sigmoid(x) -->\n","     <g transform=\"translate(14.798438 101.671875)rotate(-90)scale(0.1 -0.1)\">\n","      <defs>\n","       <path id=\"DejaVuSans-73\" d=\"M 2834 3397 \n","L 2834 2853 \n","Q 2591 2978 2328 3040 \n","Q 2066 3103 1784 3103 \n","Q 1356 3103 1142 2972 \n","Q 928 2841 928 2578 \n","Q 928 2378 1081 2264 \n","Q 1234 2150 1697 2047 \n","L 1894 2003 \n","Q 2506 1872 2764 1633 \n","Q 3022 1394 3022 966 \n","Q 3022 478 2636 193 \n","Q 2250 -91 1575 -91 \n","Q 1294 -91 989 -36 \n","Q 684 19 347 128 \n","L 347 722 \n","Q 666 556 975 473 \n","Q 1284 391 1588 391 \n","Q 1994 391 2212 530 \n","Q 2431 669 2431 922 \n","Q 2431 1156 2273 1281 \n","Q 2116 1406 1581 1522 \n","L 1381 1569 \n","Q 847 1681 609 1914 \n","Q 372 2147 372 2553 \n","Q 372 3047 722 3315 \n","Q 1072 3584 1716 3584 \n","Q 2034 3584 2315 3537 \n","Q 2597 3491 2834 3397 \n","z\n","\" transform=\"scale(0.015625)\"/>\n","       <path id=\"DejaVuSans-69\" d=\"M 603 3500 \n","L 1178 3500 \n","L 1178 0 \n","L 603 0 \n","L 603 3500 \n","z\n","M 603 4863 \n","L 1178 4863 \n","L 1178 4134 \n","L 603 4134 \n","L 603 4863 \n","z\n","\" transform=\"scale(0.015625)\"/>\n","       <path id=\"DejaVuSans-67\" d=\"M 2906 1791 \n","Q 2906 2416 2648 2759 \n","Q 2391 3103 1925 3103 \n","Q 1463 3103 1205 2759 \n","Q 947 2416 947 1791 \n","Q 947 1169 1205 825 \n","Q 1463 481 1925 481 \n","Q 2391 481 2648 825 \n","Q 2906 1169 2906 1791 \n","z\n","M 3481 434 \n","Q 3481 -459 3084 -895 \n","Q 2688 -1331 1869 -1331 \n","Q 1566 -1331 1297 -1286 \n","Q 1028 -1241 775 -1147 \n","L 775 -588 \n","Q 1028 -725 1275 -790 \n","Q 1522 -856 1778 -856 \n","Q 2344 -856 2625 -561 \n","Q 2906 -266 2906 331 \n","L 2906 616 \n","Q 2728 306 2450 153 \n","Q 2172 0 1784 0 \n","Q 1141 0 747 490 \n","Q 353 981 353 1791 \n","Q 353 2603 747 3093 \n","Q 1141 3584 1784 3584 \n","Q 2172 3584 2450 3431 \n","Q 2728 3278 2906 2969 \n","L 2906 3500 \n","L 3481 3500 \n","L 3481 434 \n","z\n","\" transform=\"scale(0.015625)\"/>\n","       <path id=\"DejaVuSans-6d\" d=\"M 3328 2828 \n","Q 3544 3216 3844 3400 \n","Q 4144 3584 4550 3584 \n","Q 5097 3584 5394 3201 \n","Q 5691 2819 5691 2113 \n","L 5691 0 \n","L 5113 0 \n","L 5113 2094 \n","Q 5113 2597 4934 2840 \n","Q 4756 3084 4391 3084 \n","Q 3944 3084 3684 2787 \n","Q 3425 2491 3425 1978 \n","L 3425 0 \n","L 2847 0 \n","L 2847 2094 \n","Q 2847 2600 2669 2842 \n","Q 2491 3084 2119 3084 \n","Q 1678 3084 1418 2786 \n","Q 1159 2488 1159 1978 \n","L 1159 0 \n","L 581 0 \n","L 581 3500 \n","L 1159 3500 \n","L 1159 2956 \n","Q 1356 3278 1631 3431 \n","Q 1906 3584 2284 3584 \n","Q 2666 3584 2933 3390 \n","Q 3200 3197 3328 2828 \n","z\n","\" transform=\"scale(0.015625)\"/>\n","       <path id=\"DejaVuSans-6f\" d=\"M 1959 3097 \n","Q 1497 3097 1228 2736 \n","Q 959 2375 959 1747 \n","Q 959 1119 1226 758 \n","Q 1494 397 1959 397 \n","Q 2419 397 2687 759 \n","Q 2956 1122 2956 1747 \n","Q 2956 2369 2687 2733 \n","Q 2419 3097 1959 3097 \n","z\n","M 1959 3584 \n","Q 2709 3584 3137 3096 \n","Q 3566 2609 3566 1747 \n","Q 3566 888 3137 398 \n","Q 2709 -91 1959 -91 \n","Q 1206 -91 779 398 \n","Q 353 888 353 1747 \n","Q 353 2609 779 3096 \n","Q 1206 3584 1959 3584 \n","z\n","\" transform=\"scale(0.015625)\"/>\n","       <path id=\"DejaVuSans-64\" d=\"M 2906 2969 \n","L 2906 4863 \n","L 3481 4863 \n","L 3481 0 \n","L 2906 0 \n","L 2906 525 \n","Q 2725 213 2448 61 \n","Q 2172 -91 1784 -91 \n","Q 1150 -91 751 415 \n","Q 353 922 353 1747 \n","Q 353 2572 751 3078 \n","Q 1150 3584 1784 3584 \n","Q 2172 3584 2448 3432 \n","Q 2725 3281 2906 2969 \n","z\n","M 947 1747 \n","Q 947 1113 1208 752 \n","Q 1469 391 1925 391 \n","Q 2381 391 2643 752 \n","Q 2906 1113 2906 1747 \n","Q 2906 2381 2643 2742 \n","Q 2381 3103 1925 3103 \n","Q 1469 3103 1208 2742 \n","Q 947 2381 947 1747 \n","z\n","\" transform=\"scale(0.015625)\"/>\n","       <path id=\"DejaVuSans-28\" d=\"M 1984 4856 \n","Q 1566 4138 1362 3434 \n","Q 1159 2731 1159 2009 \n","Q 1159 1288 1364 580 \n","Q 1569 -128 1984 -844 \n","L 1484 -844 \n","Q 1016 -109 783 600 \n","Q 550 1309 550 2009 \n","Q 550 2706 781 3412 \n","Q 1013 4119 1484 4856 \n","L 1984 4856 \n","z\n","\" transform=\"scale(0.015625)\"/>\n","       <path id=\"DejaVuSans-29\" d=\"M 513 4856 \n","L 1013 4856 \n","Q 1481 4119 1714 3412 \n","Q 1947 2706 1947 2009 \n","Q 1947 1309 1714 600 \n","Q 1481 -109 1013 -844 \n","L 513 -844 \n","Q 928 -128 1133 580 \n","Q 1338 1288 1338 2009 \n","Q 1338 2731 1133 3434 \n","Q 928 4138 513 4856 \n","z\n","\" transform=\"scale(0.015625)\"/>\n","      </defs>\n","      <use xlink:href=\"#DejaVuSans-73\"/>\n","      <use xlink:href=\"#DejaVuSans-69\" x=\"52.099609\"/>\n","      <use xlink:href=\"#DejaVuSans-67\" x=\"79.882812\"/>\n","      <use xlink:href=\"#DejaVuSans-6d\" x=\"143.359375\"/>\n","      <use xlink:href=\"#DejaVuSans-6f\" x=\"240.771484\"/>\n","      <use xlink:href=\"#DejaVuSans-69\" x=\"301.953125\"/>\n","      <use xlink:href=\"#DejaVuSans-64\" x=\"329.736328\"/>\n","      <use xlink:href=\"#DejaVuSans-28\" x=\"393.212891\"/>\n","      <use xlink:href=\"#DejaVuSans-78\" x=\"432.226562\"/>\n","      <use xlink:href=\"#DejaVuSans-29\" x=\"491.40625\"/>\n","     </g>\n","    </g>\n","   </g>\n","   <g id=\"line2d_31\">\n","    <path d=\"M 56.463068 136.922727 \n","L 83.581419 136.737576 \n","L 96.343001 136.460985 \n","L 105.914183 136.050351 \n","L 113.890169 135.464716 \n","L 120.27096 134.740503 \n","L 125.056551 133.981298 \n","L 129.842142 132.971411 \n","L 133.032538 132.122022 \n","L 136.222933 131.100797 \n","L 139.413328 129.877043 \n","L 142.60372 128.416417 \n","L 145.794115 126.68132 \n","L 148.98451 124.631763 \n","L 152.174906 122.226805 \n","L 155.365299 119.426754 \n","L 158.555695 116.196162 \n","L 161.74609 112.507641 \n","L 164.936483 108.346273 \n","L 168.126879 103.714222 \n","L 171.317273 98.634885 \n","L 174.507668 93.155705 \n","L 179.29326 84.35135 \n","L 193.650036 57.139934 \n","L 196.84043 51.660754 \n","L 200.030825 46.581413 \n","L 203.22122 41.949361 \n","L 206.411613 37.787994 \n","L 209.602009 34.09948 \n","L 212.792404 30.868895 \n","L 215.982798 28.068841 \n","L 219.173193 25.663874 \n","L 222.363588 23.614318 \n","L 225.553984 21.879222 \n","L 228.744379 20.418596 \n","L 231.934771 19.194841 \n","L 235.125166 18.173619 \n","L 239.910757 16.955406 \n","L 244.696348 16.036702 \n","L 249.481939 15.346978 \n","L 255.86273 14.689795 \n","L 263.838716 14.158904 \n","L 273.409899 13.786942 \n","L 286.17148 13.536533 \n","L 306.90904 13.38742 \n","L 310.099432 13.377273 \n","L 310.099432 13.377273 \n","\" clip-path=\"url(#p178e410158)\" style=\"fill: none; stroke: #1f77b4; stroke-width: 1.5; stroke-linecap: square\"/>\n","   </g>\n","   <g id=\"patch_3\">\n","    <path d=\"M 43.78125 143.1 \n","L 43.78125 7.2 \n","\" style=\"fill: none; stroke: #000000; stroke-width: 0.8; stroke-linejoin: miter; stroke-linecap: square\"/>\n","   </g>\n","   <g id=\"patch_4\">\n","    <path d=\"M 322.78125 143.1 \n","L 322.78125 7.2 \n","\" style=\"fill: none; stroke: #000000; stroke-width: 0.8; stroke-linejoin: miter; stroke-linecap: square\"/>\n","   </g>\n","   <g id=\"patch_5\">\n","    <path d=\"M 43.78125 143.1 \n","L 322.78125 143.1 \n","\" style=\"fill: none; stroke: #000000; stroke-width: 0.8; stroke-linejoin: miter; stroke-linecap: square\"/>\n","   </g>\n","   <g id=\"patch_6\">\n","    <path d=\"M 43.78125 7.2 \n","L 322.78125 7.2 \n","\" style=\"fill: none; stroke: #000000; stroke-width: 0.8; stroke-linejoin: miter; stroke-linecap: square\"/>\n","   </g>\n","  </g>\n"," </g>\n"," <defs>\n","  <clipPath id=\"p178e410158\">\n","   <rect x=\"43.78125\" y=\"7.2\" width=\"279\" height=\"135.9\"/>\n","  </clipPath>\n"," </defs>\n","</svg>\n"],"text/plain":["<Figure size 360x180 with 1 Axes>"]},"metadata":{"needs_background":"light"},"output_type":"display_data"}],"source":["y = torch.sigmoid(x)\n","d2l.plot(x.detach(), y.detach(), 'x', 'sigmoid(x)', figsize=(5, 2.5))"]},{"cell_type":"markdown","id":"69b8bbd7","metadata":{"origin_pos":16,"id":"69b8bbd7"},"source":["sigmoid函数的导数为下面的公式：\n","\n","$$\\frac{d}{dx} \\operatorname{sigmoid}(x) = \\frac{\\exp(-x)}{(1 + \\exp(-x))^2} = \\operatorname{sigmoid}(x)\\left(1-\\operatorname{sigmoid}(x)\\right).$$\n","\n","sigmoid函数的导数图像如下所示。\n","注意，当输入为0时，sigmoid函数的导数达到最大值0.25；\n","而输入在任一方向上越远离0点时，导数越接近0。\n"]},{"cell_type":"code","execution_count":null,"id":"1f28011c","metadata":{"execution":{"iopub.execute_input":"2022-07-31T22:40:54.584686Z","iopub.status.busy":"2022-07-31T22:40:54.584070Z","iopub.status.idle":"2022-07-31T22:40:54.784496Z","shell.execute_reply":"2022-07-31T22:40:54.783858Z"},"origin_pos":18,"tab":["pytorch"],"id":"1f28011c","outputId":"622a3e87-c0c3-4d3b-bad1-dc073cc71aca"},"outputs":[{"data":{"image/svg+xml":["<?xml version=\"1.0\" encoding=\"utf-8\" standalone=\"no\"?>\n","<!DOCTYPE svg PUBLIC \"-//W3C//DTD SVG 1.1//EN\"\n","  \"http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd\">\n","<svg xmlns:xlink=\"http://www.w3.org/1999/xlink\" width=\"336.34375pt\" height=\"180.65625pt\" viewBox=\"0 0 336.34375 180.65625\" xmlns=\"http://www.w3.org/2000/svg\" version=\"1.1\">\n"," <metadata>\n","  <rdf:RDF xmlns:dc=\"http://purl.org/dc/elements/1.1/\" xmlns:cc=\"http://creativecommons.org/ns#\" xmlns:rdf=\"http://www.w3.org/1999/02/22-rdf-syntax-ns#\">\n","   <cc:Work>\n","    <dc:type rdf:resource=\"http://purl.org/dc/dcmitype/StillImage\"/>\n","    <dc:date>2022-07-31T22:40:54.740402</dc:date>\n","    <dc:format>image/svg+xml</dc:format>\n","    <dc:creator>\n","     <cc:Agent>\n","      <dc:title>Matplotlib v3.5.1, https://matplotlib.org/</dc:title>\n","     </cc:Agent>\n","    </dc:creator>\n","   </cc:Work>\n","  </rdf:RDF>\n"," </metadata>\n"," <defs>\n","  <style type=\"text/css\">*{stroke-linejoin: round; stroke-linecap: butt}</style>\n"," </defs>\n"," <g id=\"figure_1\">\n","  <g id=\"patch_1\">\n","   <path d=\"M 0 180.65625 \n","L 336.34375 180.65625 \n","L 336.34375 0 \n","L 0 0 \n","L 0 180.65625 \n","z\n","\" style=\"fill: none\"/>\n","  </g>\n","  <g id=\"axes_1\">\n","   <g id=\"patch_2\">\n","    <path d=\"M 50.14375 143.1 \n","L 329.14375 143.1 \n","L 329.14375 7.2 \n","L 50.14375 7.2 \n","z\n","\" style=\"fill: #ffffff\"/>\n","   </g>\n","   <g id=\"matplotlib.axis_1\">\n","    <g id=\"xtick_1\">\n","     <g id=\"line2d_1\">\n","      <path d=\"M 62.825568 143.1 \n","L 62.825568 7.2 \n","\" clip-path=\"url(#p8780931454)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n","     </g>\n","     <g id=\"line2d_2\">\n","      <defs>\n","       <path id=\"mbdaad5c070\" d=\"M 0 0 \n","L 0 3.5 \n","\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n","      </defs>\n","      <g>\n","       <use xlink:href=\"#mbdaad5c070\" x=\"62.825568\" y=\"143.1\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n","      </g>\n","     </g>\n","     <g id=\"text_1\">\n","      <!-- −8 -->\n","      <g transform=\"translate(55.454474 157.698438)scale(0.1 -0.1)\">\n","       <defs>\n","        <path id=\"DejaVuSans-2212\" d=\"M 678 2272 \n","L 4684 2272 \n","L 4684 1741 \n","L 678 1741 \n","L 678 2272 \n","z\n","\" transform=\"scale(0.015625)\"/>\n","        <path id=\"DejaVuSans-38\" d=\"M 2034 2216 \n","Q 1584 2216 1326 1975 \n","Q 1069 1734 1069 1313 \n","Q 1069 891 1326 650 \n","Q 1584 409 2034 409 \n","Q 2484 409 2743 651 \n","Q 3003 894 3003 1313 \n","Q 3003 1734 2745 1975 \n","Q 2488 2216 2034 2216 \n","z\n","M 1403 2484 \n","Q 997 2584 770 2862 \n","Q 544 3141 544 3541 \n","Q 544 4100 942 4425 \n","Q 1341 4750 2034 4750 \n","Q 2731 4750 3128 4425 \n","Q 3525 4100 3525 3541 \n","Q 3525 3141 3298 2862 \n","Q 3072 2584 2669 2484 \n","Q 3125 2378 3379 2068 \n","Q 3634 1759 3634 1313 \n","Q 3634 634 3220 271 \n","Q 2806 -91 2034 -91 \n","Q 1263 -91 848 271 \n","Q 434 634 434 1313 \n","Q 434 1759 690 2068 \n","Q 947 2378 1403 2484 \n","z\n","M 1172 3481 \n","Q 1172 3119 1398 2916 \n","Q 1625 2713 2034 2713 \n","Q 2441 2713 2670 2916 \n","Q 2900 3119 2900 3481 \n","Q 2900 3844 2670 4047 \n","Q 2441 4250 2034 4250 \n","Q 1625 4250 1398 4047 \n","Q 1172 3844 1172 3481 \n","z\n","\" transform=\"scale(0.015625)\"/>\n","       </defs>\n","       <use xlink:href=\"#DejaVuSans-2212\"/>\n","       <use xlink:href=\"#DejaVuSans-38\" x=\"83.789062\"/>\n","      </g>\n","     </g>\n","    </g>\n","    <g id=\"xtick_2\">\n","     <g id=\"line2d_3\">\n","      <path d=\"M 94.729514 143.1 \n","L 94.729514 7.2 \n","\" clip-path=\"url(#p8780931454)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n","     </g>\n","     <g id=\"line2d_4\">\n","      <g>\n","       <use xlink:href=\"#mbdaad5c070\" x=\"94.729514\" y=\"143.1\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n","      </g>\n","     </g>\n","     <g id=\"text_2\">\n","      <!-- −6 -->\n","      <g transform=\"translate(87.35842 157.698438)scale(0.1 -0.1)\">\n","       <defs>\n","        <path id=\"DejaVuSans-36\" d=\"M 2113 2584 \n","Q 1688 2584 1439 2293 \n","Q 1191 2003 1191 1497 \n","Q 1191 994 1439 701 \n","Q 1688 409 2113 409 \n","Q 2538 409 2786 701 \n","Q 3034 994 3034 1497 \n","Q 3034 2003 2786 2293 \n","Q 2538 2584 2113 2584 \n","z\n","M 3366 4563 \n","L 3366 3988 \n","Q 3128 4100 2886 4159 \n","Q 2644 4219 2406 4219 \n","Q 1781 4219 1451 3797 \n","Q 1122 3375 1075 2522 \n","Q 1259 2794 1537 2939 \n","Q 1816 3084 2150 3084 \n","Q 2853 3084 3261 2657 \n","Q 3669 2231 3669 1497 \n","Q 3669 778 3244 343 \n","Q 2819 -91 2113 -91 \n","Q 1303 -91 875 529 \n","Q 447 1150 447 2328 \n","Q 447 3434 972 4092 \n","Q 1497 4750 2381 4750 \n","Q 2619 4750 2861 4703 \n","Q 3103 4656 3366 4563 \n","z\n","\" transform=\"scale(0.015625)\"/>\n","       </defs>\n","       <use xlink:href=\"#DejaVuSans-2212\"/>\n","       <use xlink:href=\"#DejaVuSans-36\" x=\"83.789062\"/>\n","      </g>\n","     </g>\n","    </g>\n","    <g id=\"xtick_3\">\n","     <g id=\"line2d_5\">\n","      <path d=\"M 126.63346 143.1 \n","L 126.63346 7.2 \n","\" clip-path=\"url(#p8780931454)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n","     </g>\n","     <g id=\"line2d_6\">\n","      <g>\n","       <use xlink:href=\"#mbdaad5c070\" x=\"126.63346\" y=\"143.1\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n","      </g>\n","     </g>\n","     <g id=\"text_3\">\n","      <!-- −4 -->\n","      <g transform=\"translate(119.262366 157.698438)scale(0.1 -0.1)\">\n","       <defs>\n","        <path id=\"DejaVuSans-34\" d=\"M 2419 4116 \n","L 825 1625 \n","L 2419 1625 \n","L 2419 4116 \n","z\n","M 2253 4666 \n","L 3047 4666 \n","L 3047 1625 \n","L 3713 1625 \n","L 3713 1100 \n","L 3047 1100 \n","L 3047 0 \n","L 2419 0 \n","L 2419 1100 \n","L 313 1100 \n","L 313 1709 \n","L 2253 4666 \n","z\n","\" transform=\"scale(0.015625)\"/>\n","       </defs>\n","       <use xlink:href=\"#DejaVuSans-2212\"/>\n","       <use xlink:href=\"#DejaVuSans-34\" x=\"83.789062\"/>\n","      </g>\n","     </g>\n","    </g>\n","    <g id=\"xtick_4\">\n","     <g id=\"line2d_7\">\n","      <path d=\"M 158.537406 143.1 \n","L 158.537406 7.2 \n","\" clip-path=\"url(#p8780931454)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n","     </g>\n","     <g id=\"line2d_8\">\n","      <g>\n","       <use xlink:href=\"#mbdaad5c070\" x=\"158.537406\" y=\"143.1\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n","      </g>\n","     </g>\n","     <g id=\"text_4\">\n","      <!-- −2 -->\n","      <g transform=\"translate(151.166312 157.698438)scale(0.1 -0.1)\">\n","       <defs>\n","        <path id=\"DejaVuSans-32\" d=\"M 1228 531 \n","L 3431 531 \n","L 3431 0 \n","L 469 0 \n","L 469 531 \n","Q 828 903 1448 1529 \n","Q 2069 2156 2228 2338 \n","Q 2531 2678 2651 2914 \n","Q 2772 3150 2772 3378 \n","Q 2772 3750 2511 3984 \n","Q 2250 4219 1831 4219 \n","Q 1534 4219 1204 4116 \n","Q 875 4013 500 3803 \n","L 500 4441 \n","Q 881 4594 1212 4672 \n","Q 1544 4750 1819 4750 \n","Q 2544 4750 2975 4387 \n","Q 3406 4025 3406 3419 \n","Q 3406 3131 3298 2873 \n","Q 3191 2616 2906 2266 \n","Q 2828 2175 2409 1742 \n","Q 1991 1309 1228 531 \n","z\n","\" transform=\"scale(0.015625)\"/>\n","       </defs>\n","       <use xlink:href=\"#DejaVuSans-2212\"/>\n","       <use xlink:href=\"#DejaVuSans-32\" x=\"83.789062\"/>\n","      </g>\n","     </g>\n","    </g>\n","    <g id=\"xtick_5\">\n","     <g id=\"line2d_9\">\n","      <path d=\"M 190.441352 143.1 \n","L 190.441352 7.2 \n","\" clip-path=\"url(#p8780931454)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n","     </g>\n","     <g id=\"line2d_10\">\n","      <g>\n","       <use xlink:href=\"#mbdaad5c070\" x=\"190.441352\" y=\"143.1\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n","      </g>\n","     </g>\n","     <g id=\"text_5\">\n","      <!-- 0 -->\n","      <g transform=\"translate(187.260102 157.698438)scale(0.1 -0.1)\">\n","       <defs>\n","        <path id=\"DejaVuSans-30\" d=\"M 2034 4250 \n","Q 1547 4250 1301 3770 \n","Q 1056 3291 1056 2328 \n","Q 1056 1369 1301 889 \n","Q 1547 409 2034 409 \n","Q 2525 409 2770 889 \n","Q 3016 1369 3016 2328 \n","Q 3016 3291 2770 3770 \n","Q 2525 4250 2034 4250 \n","z\n","M 2034 4750 \n","Q 2819 4750 3233 4129 \n","Q 3647 3509 3647 2328 \n","Q 3647 1150 3233 529 \n","Q 2819 -91 2034 -91 \n","Q 1250 -91 836 529 \n","Q 422 1150 422 2328 \n","Q 422 3509 836 4129 \n","Q 1250 4750 2034 4750 \n","z\n","\" transform=\"scale(0.015625)\"/>\n","       </defs>\n","       <use xlink:href=\"#DejaVuSans-30\"/>\n","      </g>\n","     </g>\n","    </g>\n","    <g id=\"xtick_6\">\n","     <g id=\"line2d_11\">\n","      <path d=\"M 222.345298 143.1 \n","L 222.345298 7.2 \n","\" clip-path=\"url(#p8780931454)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n","     </g>\n","     <g id=\"line2d_12\">\n","      <g>\n","       <use xlink:href=\"#mbdaad5c070\" x=\"222.345298\" y=\"143.1\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n","      </g>\n","     </g>\n","     <g id=\"text_6\">\n","      <!-- 2 -->\n","      <g transform=\"translate(219.164048 157.698438)scale(0.1 -0.1)\">\n","       <use xlink:href=\"#DejaVuSans-32\"/>\n","      </g>\n","     </g>\n","    </g>\n","    <g id=\"xtick_7\">\n","     <g id=\"line2d_13\">\n","      <path d=\"M 254.249243 143.1 \n","L 254.249243 7.2 \n","\" clip-path=\"url(#p8780931454)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n","     </g>\n","     <g id=\"line2d_14\">\n","      <g>\n","       <use xlink:href=\"#mbdaad5c070\" x=\"254.249243\" y=\"143.1\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n","      </g>\n","     </g>\n","     <g id=\"text_7\">\n","      <!-- 4 -->\n","      <g transform=\"translate(251.067993 157.698438)scale(0.1 -0.1)\">\n","       <use xlink:href=\"#DejaVuSans-34\"/>\n","      </g>\n","     </g>\n","    </g>\n","    <g id=\"xtick_8\">\n","     <g id=\"line2d_15\">\n","      <path d=\"M 286.153189 143.1 \n","L 286.153189 7.2 \n","\" clip-path=\"url(#p8780931454)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n","     </g>\n","     <g id=\"line2d_16\">\n","      <g>\n","       <use xlink:href=\"#mbdaad5c070\" x=\"286.153189\" y=\"143.1\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n","      </g>\n","     </g>\n","     <g id=\"text_8\">\n","      <!-- 6 -->\n","      <g transform=\"translate(282.971939 157.698438)scale(0.1 -0.1)\">\n","       <use xlink:href=\"#DejaVuSans-36\"/>\n","      </g>\n","     </g>\n","    </g>\n","    <g id=\"xtick_9\">\n","     <g id=\"line2d_17\">\n","      <path d=\"M 318.057135 143.1 \n","L 318.057135 7.2 \n","\" clip-path=\"url(#p8780931454)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n","     </g>\n","     <g id=\"line2d_18\">\n","      <g>\n","       <use xlink:href=\"#mbdaad5c070\" x=\"318.057135\" y=\"143.1\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n","      </g>\n","     </g>\n","     <g id=\"text_9\">\n","      <!-- 8 -->\n","      <g transform=\"translate(314.875885 157.698438)scale(0.1 -0.1)\">\n","       <use xlink:href=\"#DejaVuSans-38\"/>\n","      </g>\n","     </g>\n","    </g>\n","    <g id=\"text_10\">\n","     <!-- x -->\n","     <g transform=\"translate(186.684375 171.376563)scale(0.1 -0.1)\">\n","      <defs>\n","       <path id=\"DejaVuSans-78\" d=\"M 3513 3500 \n","L 2247 1797 \n","L 3578 0 \n","L 2900 0 \n","L 1881 1375 \n","L 863 0 \n","L 184 0 \n","L 1544 1831 \n","L 300 3500 \n","L 978 3500 \n","L 1906 2253 \n","L 2834 3500 \n","L 3513 3500 \n","z\n","\" transform=\"scale(0.015625)\"/>\n","      </defs>\n","      <use xlink:href=\"#DejaVuSans-78\"/>\n","     </g>\n","    </g>\n","   </g>\n","   <g id=\"matplotlib.axis_2\">\n","    <g id=\"ytick_1\">\n","     <g id=\"line2d_19\">\n","      <path d=\"M 50.14375 137.088618 \n","L 329.14375 137.088618 \n","\" clip-path=\"url(#p8780931454)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n","     </g>\n","     <g id=\"line2d_20\">\n","      <defs>\n","       <path id=\"mc4f95a77e5\" d=\"M 0 0 \n","L -3.5 0 \n","\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n","      </defs>\n","      <g>\n","       <use xlink:href=\"#mc4f95a77e5\" x=\"50.14375\" y=\"137.088618\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n","      </g>\n","     </g>\n","     <g id=\"text_11\">\n","      <!-- 0.00 -->\n","      <g transform=\"translate(20.878125 140.887837)scale(0.1 -0.1)\">\n","       <defs>\n","        <path id=\"DejaVuSans-2e\" d=\"M 684 794 \n","L 1344 794 \n","L 1344 0 \n","L 684 0 \n","L 684 794 \n","z\n","\" transform=\"scale(0.015625)\"/>\n","       </defs>\n","       <use xlink:href=\"#DejaVuSans-30\"/>\n","       <use xlink:href=\"#DejaVuSans-2e\" x=\"63.623047\"/>\n","       <use xlink:href=\"#DejaVuSans-30\" x=\"95.410156\"/>\n","       <use xlink:href=\"#DejaVuSans-30\" x=\"159.033203\"/>\n","      </g>\n","     </g>\n","    </g>\n","    <g id=\"ytick_2\">\n","     <g id=\"line2d_21\">\n","      <path d=\"M 50.14375 112.346349 \n","L 329.14375 112.346349 \n","\" clip-path=\"url(#p8780931454)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n","     </g>\n","     <g id=\"line2d_22\">\n","      <g>\n","       <use xlink:href=\"#mc4f95a77e5\" x=\"50.14375\" y=\"112.346349\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n","      </g>\n","     </g>\n","     <g id=\"text_12\">\n","      <!-- 0.05 -->\n","      <g transform=\"translate(20.878125 116.145568)scale(0.1 -0.1)\">\n","       <defs>\n","        <path id=\"DejaVuSans-35\" d=\"M 691 4666 \n","L 3169 4666 \n","L 3169 4134 \n","L 1269 4134 \n","L 1269 2991 \n","Q 1406 3038 1543 3061 \n","Q 1681 3084 1819 3084 \n","Q 2600 3084 3056 2656 \n","Q 3513 2228 3513 1497 \n","Q 3513 744 3044 326 \n","Q 2575 -91 1722 -91 \n","Q 1428 -91 1123 -41 \n","Q 819 9 494 109 \n","L 494 744 \n","Q 775 591 1075 516 \n","Q 1375 441 1709 441 \n","Q 2250 441 2565 725 \n","Q 2881 1009 2881 1497 \n","Q 2881 1984 2565 2268 \n","Q 2250 2553 1709 2553 \n","Q 1456 2553 1204 2497 \n","Q 953 2441 691 2322 \n","L 691 4666 \n","z\n","\" transform=\"scale(0.015625)\"/>\n","       </defs>\n","       <use xlink:href=\"#DejaVuSans-30\"/>\n","       <use xlink:href=\"#DejaVuSans-2e\" x=\"63.623047\"/>\n","       <use xlink:href=\"#DejaVuSans-30\" x=\"95.410156\"/>\n","       <use xlink:href=\"#DejaVuSans-35\" x=\"159.033203\"/>\n","      </g>\n","     </g>\n","    </g>\n","    <g id=\"ytick_3\">\n","     <g id=\"line2d_23\">\n","      <path d=\"M 50.14375 87.60408 \n","L 329.14375 87.60408 \n","\" clip-path=\"url(#p8780931454)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n","     </g>\n","     <g id=\"line2d_24\">\n","      <g>\n","       <use xlink:href=\"#mc4f95a77e5\" x=\"50.14375\" y=\"87.60408\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n","      </g>\n","     </g>\n","     <g id=\"text_13\">\n","      <!-- 0.10 -->\n","      <g transform=\"translate(20.878125 91.403299)scale(0.1 -0.1)\">\n","       <defs>\n","        <path id=\"DejaVuSans-31\" d=\"M 794 531 \n","L 1825 531 \n","L 1825 4091 \n","L 703 3866 \n","L 703 4441 \n","L 1819 4666 \n","L 2450 4666 \n","L 2450 531 \n","L 3481 531 \n","L 3481 0 \n","L 794 0 \n","L 794 531 \n","z\n","\" transform=\"scale(0.015625)\"/>\n","       </defs>\n","       <use xlink:href=\"#DejaVuSans-30\"/>\n","       <use xlink:href=\"#DejaVuSans-2e\" x=\"63.623047\"/>\n","       <use xlink:href=\"#DejaVuSans-31\" x=\"95.410156\"/>\n","       <use xlink:href=\"#DejaVuSans-30\" x=\"159.033203\"/>\n","      </g>\n","     </g>\n","    </g>\n","    <g id=\"ytick_4\">\n","     <g id=\"line2d_25\">\n","      <path d=\"M 50.14375 62.861811 \n","L 329.14375 62.861811 \n","\" clip-path=\"url(#p8780931454)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n","     </g>\n","     <g id=\"line2d_26\">\n","      <g>\n","       <use xlink:href=\"#mc4f95a77e5\" x=\"50.14375\" y=\"62.861811\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n","      </g>\n","     </g>\n","     <g id=\"text_14\">\n","      <!-- 0.15 -->\n","      <g transform=\"translate(20.878125 66.66103)scale(0.1 -0.1)\">\n","       <use xlink:href=\"#DejaVuSans-30\"/>\n","       <use xlink:href=\"#DejaVuSans-2e\" x=\"63.623047\"/>\n","       <use xlink:href=\"#DejaVuSans-31\" x=\"95.410156\"/>\n","       <use xlink:href=\"#DejaVuSans-35\" x=\"159.033203\"/>\n","      </g>\n","     </g>\n","    </g>\n","    <g id=\"ytick_5\">\n","     <g id=\"line2d_27\">\n","      <path d=\"M 50.14375 38.119542 \n","L 329.14375 38.119542 \n","\" clip-path=\"url(#p8780931454)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n","     </g>\n","     <g id=\"line2d_28\">\n","      <g>\n","       <use xlink:href=\"#mc4f95a77e5\" x=\"50.14375\" y=\"38.119542\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n","      </g>\n","     </g>\n","     <g id=\"text_15\">\n","      <!-- 0.20 -->\n","      <g transform=\"translate(20.878125 41.918761)scale(0.1 -0.1)\">\n","       <use xlink:href=\"#DejaVuSans-30\"/>\n","       <use xlink:href=\"#DejaVuSans-2e\" x=\"63.623047\"/>\n","       <use xlink:href=\"#DejaVuSans-32\" x=\"95.410156\"/>\n","       <use xlink:href=\"#DejaVuSans-30\" x=\"159.033203\"/>\n","      </g>\n","     </g>\n","    </g>\n","    <g id=\"ytick_6\">\n","     <g id=\"line2d_29\">\n","      <path d=\"M 50.14375 13.377273 \n","L 329.14375 13.377273 \n","\" clip-path=\"url(#p8780931454)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n","     </g>\n","     <g id=\"line2d_30\">\n","      <g>\n","       <use xlink:href=\"#mc4f95a77e5\" x=\"50.14375\" y=\"13.377273\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n","      </g>\n","     </g>\n","     <g id=\"text_16\">\n","      <!-- 0.25 -->\n","      <g transform=\"translate(20.878125 17.176491)scale(0.1 -0.1)\">\n","       <use xlink:href=\"#DejaVuSans-30\"/>\n","       <use xlink:href=\"#DejaVuSans-2e\" x=\"63.623047\"/>\n","       <use xlink:href=\"#DejaVuSans-32\" x=\"95.410156\"/>\n","       <use xlink:href=\"#DejaVuSans-35\" x=\"159.033203\"/>\n","      </g>\n","     </g>\n","    </g>\n","    <g id=\"text_17\">\n","     <!-- grad of sigmoid -->\n","     <g transform=\"translate(14.798438 114.276562)rotate(-90)scale(0.1 -0.1)\">\n","      <defs>\n","       <path id=\"DejaVuSans-67\" d=\"M 2906 1791 \n","Q 2906 2416 2648 2759 \n","Q 2391 3103 1925 3103 \n","Q 1463 3103 1205 2759 \n","Q 947 2416 947 1791 \n","Q 947 1169 1205 825 \n","Q 1463 481 1925 481 \n","Q 2391 481 2648 825 \n","Q 2906 1169 2906 1791 \n","z\n","M 3481 434 \n","Q 3481 -459 3084 -895 \n","Q 2688 -1331 1869 -1331 \n","Q 1566 -1331 1297 -1286 \n","Q 1028 -1241 775 -1147 \n","L 775 -588 \n","Q 1028 -725 1275 -790 \n","Q 1522 -856 1778 -856 \n","Q 2344 -856 2625 -561 \n","Q 2906 -266 2906 331 \n","L 2906 616 \n","Q 2728 306 2450 153 \n","Q 2172 0 1784 0 \n","Q 1141 0 747 490 \n","Q 353 981 353 1791 \n","Q 353 2603 747 3093 \n","Q 1141 3584 1784 3584 \n","Q 2172 3584 2450 3431 \n","Q 2728 3278 2906 2969 \n","L 2906 3500 \n","L 3481 3500 \n","L 3481 434 \n","z\n","\" transform=\"scale(0.015625)\"/>\n","       <path id=\"DejaVuSans-72\" d=\"M 2631 2963 \n","Q 2534 3019 2420 3045 \n","Q 2306 3072 2169 3072 \n","Q 1681 3072 1420 2755 \n","Q 1159 2438 1159 1844 \n","L 1159 0 \n","L 581 0 \n","L 581 3500 \n","L 1159 3500 \n","L 1159 2956 \n","Q 1341 3275 1631 3429 \n","Q 1922 3584 2338 3584 \n","Q 2397 3584 2469 3576 \n","Q 2541 3569 2628 3553 \n","L 2631 2963 \n","z\n","\" transform=\"scale(0.015625)\"/>\n","       <path id=\"DejaVuSans-61\" d=\"M 2194 1759 \n","Q 1497 1759 1228 1600 \n","Q 959 1441 959 1056 \n","Q 959 750 1161 570 \n","Q 1363 391 1709 391 \n","Q 2188 391 2477 730 \n","Q 2766 1069 2766 1631 \n","L 2766 1759 \n","L 2194 1759 \n","z\n","M 3341 1997 \n","L 3341 0 \n","L 2766 0 \n","L 2766 531 \n","Q 2569 213 2275 61 \n","Q 1981 -91 1556 -91 \n","Q 1019 -91 701 211 \n","Q 384 513 384 1019 \n","Q 384 1609 779 1909 \n","Q 1175 2209 1959 2209 \n","L 2766 2209 \n","L 2766 2266 \n","Q 2766 2663 2505 2880 \n","Q 2244 3097 1772 3097 \n","Q 1472 3097 1187 3025 \n","Q 903 2953 641 2809 \n","L 641 3341 \n","Q 956 3463 1253 3523 \n","Q 1550 3584 1831 3584 \n","Q 2591 3584 2966 3190 \n","Q 3341 2797 3341 1997 \n","z\n","\" transform=\"scale(0.015625)\"/>\n","       <path id=\"DejaVuSans-64\" d=\"M 2906 2969 \n","L 2906 4863 \n","L 3481 4863 \n","L 3481 0 \n","L 2906 0 \n","L 2906 525 \n","Q 2725 213 2448 61 \n","Q 2172 -91 1784 -91 \n","Q 1150 -91 751 415 \n","Q 353 922 353 1747 \n","Q 353 2572 751 3078 \n","Q 1150 3584 1784 3584 \n","Q 2172 3584 2448 3432 \n","Q 2725 3281 2906 2969 \n","z\n","M 947 1747 \n","Q 947 1113 1208 752 \n","Q 1469 391 1925 391 \n","Q 2381 391 2643 752 \n","Q 2906 1113 2906 1747 \n","Q 2906 2381 2643 2742 \n","Q 2381 3103 1925 3103 \n","Q 1469 3103 1208 2742 \n","Q 947 2381 947 1747 \n","z\n","\" transform=\"scale(0.015625)\"/>\n","       <path id=\"DejaVuSans-20\" transform=\"scale(0.015625)\"/>\n","       <path id=\"DejaVuSans-6f\" d=\"M 1959 3097 \n","Q 1497 3097 1228 2736 \n","Q 959 2375 959 1747 \n","Q 959 1119 1226 758 \n","Q 1494 397 1959 397 \n","Q 2419 397 2687 759 \n","Q 2956 1122 2956 1747 \n","Q 2956 2369 2687 2733 \n","Q 2419 3097 1959 3097 \n","z\n","M 1959 3584 \n","Q 2709 3584 3137 3096 \n","Q 3566 2609 3566 1747 \n","Q 3566 888 3137 398 \n","Q 2709 -91 1959 -91 \n","Q 1206 -91 779 398 \n","Q 353 888 353 1747 \n","Q 353 2609 779 3096 \n","Q 1206 3584 1959 3584 \n","z\n","\" transform=\"scale(0.015625)\"/>\n","       <path id=\"DejaVuSans-66\" d=\"M 2375 4863 \n","L 2375 4384 \n","L 1825 4384 \n","Q 1516 4384 1395 4259 \n","Q 1275 4134 1275 3809 \n","L 1275 3500 \n","L 2222 3500 \n","L 2222 3053 \n","L 1275 3053 \n","L 1275 0 \n","L 697 0 \n","L 697 3053 \n","L 147 3053 \n","L 147 3500 \n","L 697 3500 \n","L 697 3744 \n","Q 697 4328 969 4595 \n","Q 1241 4863 1831 4863 \n","L 2375 4863 \n","z\n","\" transform=\"scale(0.015625)\"/>\n","       <path id=\"DejaVuSans-73\" d=\"M 2834 3397 \n","L 2834 2853 \n","Q 2591 2978 2328 3040 \n","Q 2066 3103 1784 3103 \n","Q 1356 3103 1142 2972 \n","Q 928 2841 928 2578 \n","Q 928 2378 1081 2264 \n","Q 1234 2150 1697 2047 \n","L 1894 2003 \n","Q 2506 1872 2764 1633 \n","Q 3022 1394 3022 966 \n","Q 3022 478 2636 193 \n","Q 2250 -91 1575 -91 \n","Q 1294 -91 989 -36 \n","Q 684 19 347 128 \n","L 347 722 \n","Q 666 556 975 473 \n","Q 1284 391 1588 391 \n","Q 1994 391 2212 530 \n","Q 2431 669 2431 922 \n","Q 2431 1156 2273 1281 \n","Q 2116 1406 1581 1522 \n","L 1381 1569 \n","Q 847 1681 609 1914 \n","Q 372 2147 372 2553 \n","Q 372 3047 722 3315 \n","Q 1072 3584 1716 3584 \n","Q 2034 3584 2315 3537 \n","Q 2597 3491 2834 3397 \n","z\n","\" transform=\"scale(0.015625)\"/>\n","       <path id=\"DejaVuSans-69\" d=\"M 603 3500 \n","L 1178 3500 \n","L 1178 0 \n","L 603 0 \n","L 603 3500 \n","z\n","M 603 4863 \n","L 1178 4863 \n","L 1178 4134 \n","L 603 4134 \n","L 603 4863 \n","z\n","\" transform=\"scale(0.015625)\"/>\n","       <path id=\"DejaVuSans-6d\" d=\"M 3328 2828 \n","Q 3544 3216 3844 3400 \n","Q 4144 3584 4550 3584 \n","Q 5097 3584 5394 3201 \n","Q 5691 2819 5691 2113 \n","L 5691 0 \n","L 5113 0 \n","L 5113 2094 \n","Q 5113 2597 4934 2840 \n","Q 4756 3084 4391 3084 \n","Q 3944 3084 3684 2787 \n","Q 3425 2491 3425 1978 \n","L 3425 0 \n","L 2847 0 \n","L 2847 2094 \n","Q 2847 2600 2669 2842 \n","Q 2491 3084 2119 3084 \n","Q 1678 3084 1418 2786 \n","Q 1159 2488 1159 1978 \n","L 1159 0 \n","L 581 0 \n","L 581 3500 \n","L 1159 3500 \n","L 1159 2956 \n","Q 1356 3278 1631 3431 \n","Q 1906 3584 2284 3584 \n","Q 2666 3584 2933 3390 \n","Q 3200 3197 3328 2828 \n","z\n","\" transform=\"scale(0.015625)\"/>\n","      </defs>\n","      <use xlink:href=\"#DejaVuSans-67\"/>\n","      <use xlink:href=\"#DejaVuSans-72\" x=\"63.476562\"/>\n","      <use xlink:href=\"#DejaVuSans-61\" x=\"104.589844\"/>\n","      <use xlink:href=\"#DejaVuSans-64\" x=\"165.869141\"/>\n","      <use xlink:href=\"#DejaVuSans-20\" x=\"229.345703\"/>\n","      <use xlink:href=\"#DejaVuSans-6f\" x=\"261.132812\"/>\n","      <use xlink:href=\"#DejaVuSans-66\" x=\"322.314453\"/>\n","      <use xlink:href=\"#DejaVuSans-20\" x=\"357.519531\"/>\n","      <use xlink:href=\"#DejaVuSans-73\" x=\"389.306641\"/>\n","      <use xlink:href=\"#DejaVuSans-69\" x=\"441.40625\"/>\n","      <use xlink:href=\"#DejaVuSans-67\" x=\"469.189453\"/>\n","      <use xlink:href=\"#DejaVuSans-6d\" x=\"532.666016\"/>\n","      <use xlink:href=\"#DejaVuSans-6f\" x=\"630.078125\"/>\n","      <use xlink:href=\"#DejaVuSans-69\" x=\"691.259766\"/>\n","      <use xlink:href=\"#DejaVuSans-64\" x=\"719.042969\"/>\n","     </g>\n","    </g>\n","   </g>\n","   <g id=\"line2d_31\">\n","    <path d=\"M 62.825568 136.922727 \n","L 78.777541 136.638199 \n","L 88.348723 136.26913 \n","L 96.32471 135.740413 \n","L 102.705501 135.082727 \n","L 107.491095 134.388643 \n","L 112.276683 133.457984 \n","L 115.467074 132.668631 \n","L 118.657473 131.711514 \n","L 121.847865 130.552846 \n","L 125.038257 129.15292 \n","L 128.228656 127.465526 \n","L 131.419051 125.437529 \n","L 134.609446 123.008771 \n","L 137.799842 120.112511 \n","L 140.990233 116.676734 \n","L 144.180629 112.626646 \n","L 147.371024 107.888924 \n","L 150.561419 102.39812 \n","L 153.751811 96.105742 \n","L 156.942206 88.992273 \n","L 160.132602 81.081894 \n","L 163.322997 72.45897 \n","L 168.10859 58.564138 \n","L 174.489379 39.796107 \n","L 177.679773 31.236391 \n","L 179.274971 27.375027 \n","L 180.870168 23.87579 \n","L 182.465365 20.798118 \n","L 184.060562 18.196696 \n","L 185.65576 16.119549 \n","L 187.250957 14.606178 \n","L 188.846154 13.686027 \n","L 190.441352 13.377273 \n","L 192.036549 13.686035 \n","L 193.631746 14.606185 \n","L 195.226944 16.119549 \n","L 196.822141 18.196696 \n","L 198.417338 20.798118 \n","L 200.012536 23.87579 \n","L 201.607733 27.37502 \n","L 203.20293 31.236391 \n","L 206.393325 39.796115 \n","L 211.178916 53.806965 \n","L 215.964509 67.927146 \n","L 219.154904 76.851396 \n","L 222.345298 85.133007 \n","L 225.535693 92.651352 \n","L 228.726088 99.35419 \n","L 231.916484 105.241217 \n","L 235.106879 110.348108 \n","L 238.297271 114.733159 \n","L 241.487666 118.466706 \n","L 244.678061 121.623505 \n","L 247.868457 124.277258 \n","L 251.058848 126.497666 \n","L 254.249243 128.348296 \n","L 257.439635 129.885771 \n","L 260.630034 131.159691 \n","L 263.820426 132.21301 \n","L 267.010825 133.082384 \n","L 271.79642 134.108142 \n","L 276.582007 134.873661 \n","L 282.96279 135.599466 \n","L 289.343581 136.088434 \n","L 297.319575 136.480971 \n","L 308.485945 136.786478 \n","L 316.461932 136.905286 \n","L 316.461932 136.905286 \n","\" clip-path=\"url(#p8780931454)\" style=\"fill: none; stroke: #1f77b4; stroke-width: 1.5; stroke-linecap: square\"/>\n","   </g>\n","   <g id=\"patch_3\">\n","    <path d=\"M 50.14375 143.1 \n","L 50.14375 7.2 \n","\" style=\"fill: none; stroke: #000000; stroke-width: 0.8; stroke-linejoin: miter; stroke-linecap: square\"/>\n","   </g>\n","   <g id=\"patch_4\">\n","    <path d=\"M 329.14375 143.1 \n","L 329.14375 7.2 \n","\" style=\"fill: none; stroke: #000000; stroke-width: 0.8; stroke-linejoin: miter; stroke-linecap: square\"/>\n","   </g>\n","   <g id=\"patch_5\">\n","    <path d=\"M 50.14375 143.1 \n","L 329.14375 143.1 \n","\" style=\"fill: none; stroke: #000000; stroke-width: 0.8; stroke-linejoin: miter; stroke-linecap: square\"/>\n","   </g>\n","   <g id=\"patch_6\">\n","    <path d=\"M 50.14375 7.2 \n","L 329.14375 7.2 \n","\" style=\"fill: none; stroke: #000000; stroke-width: 0.8; stroke-linejoin: miter; stroke-linecap: square\"/>\n","   </g>\n","  </g>\n"," </g>\n"," <defs>\n","  <clipPath id=\"p8780931454\">\n","   <rect x=\"50.14375\" y=\"7.2\" width=\"279\" height=\"135.9\"/>\n","  </clipPath>\n"," </defs>\n","</svg>\n"],"text/plain":["<Figure size 360x180 with 1 Axes>"]},"metadata":{"needs_background":"light"},"output_type":"display_data"}],"source":["# 清除以前的梯度\n","x.grad.data.zero_()\n","y.backward(torch.ones_like(x),retain_graph=True)\n","d2l.plot(x.detach(), x.grad, 'x', 'grad of sigmoid', figsize=(5, 2.5))"]},{"cell_type":"markdown","id":"0b847790","metadata":{"origin_pos":20,"id":"0b847790"},"source":["### tanh函数\n","\n","与sigmoid函数类似，\n","[**tanh(双曲正切)函数也能将其输入压缩转换到区间(-1, 1)上**]。\n","tanh函数的公式如下：\n","\n","(**$$\\operatorname{tanh}(x) = \\frac{1 - \\exp(-2x)}{1 + \\exp(-2x)}.$$**)\n","\n","下面我们绘制tanh函数。\n","注意，当输入在0附近时，tanh函数接近线性变换。\n","函数的形状类似于sigmoid函数，\n","不同的是tanh函数关于坐标系原点中心对称。\n"]},{"cell_type":"code","execution_count":null,"id":"64f00cb6","metadata":{"execution":{"iopub.execute_input":"2022-07-31T22:40:54.788266Z","iopub.status.busy":"2022-07-31T22:40:54.787763Z","iopub.status.idle":"2022-07-31T22:40:54.986697Z","shell.execute_reply":"2022-07-31T22:40:54.986077Z"},"origin_pos":22,"tab":["pytorch"],"id":"64f00cb6","outputId":"35d26f6c-5e90-4469-d850-5f2a0c21aa23"},"outputs":[{"data":{"image/svg+xml":["<?xml version=\"1.0\" encoding=\"utf-8\" standalone=\"no\"?>\n","<!DOCTYPE svg PUBLIC \"-//W3C//DTD SVG 1.1//EN\"\n","  \"http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd\">\n","<svg xmlns:xlink=\"http://www.w3.org/1999/xlink\" width=\"338.360937pt\" height=\"180.65625pt\" viewBox=\"0 0 338.360937 180.65625\" xmlns=\"http://www.w3.org/2000/svg\" version=\"1.1\">\n"," <metadata>\n","  <rdf:RDF xmlns:dc=\"http://purl.org/dc/elements/1.1/\" xmlns:cc=\"http://creativecommons.org/ns#\" xmlns:rdf=\"http://www.w3.org/1999/02/22-rdf-syntax-ns#\">\n","   <cc:Work>\n","    <dc:type rdf:resource=\"http://purl.org/dc/dcmitype/StillImage\"/>\n","    <dc:date>2022-07-31T22:40:54.938282</dc:date>\n","    <dc:format>image/svg+xml</dc:format>\n","    <dc:creator>\n","     <cc:Agent>\n","      <dc:title>Matplotlib v3.5.1, https://matplotlib.org/</dc:title>\n","     </cc:Agent>\n","    </dc:creator>\n","   </cc:Work>\n","  </rdf:RDF>\n"," </metadata>\n"," <defs>\n","  <style type=\"text/css\">*{stroke-linejoin: round; stroke-linecap: butt}</style>\n"," </defs>\n"," <g id=\"figure_1\">\n","  <g id=\"patch_1\">\n","   <path d=\"M 0 180.65625 \n","L 338.360937 180.65625 \n","L 338.360937 0 \n","L 0 0 \n","L 0 180.65625 \n","z\n","\" style=\"fill: none\"/>\n","  </g>\n","  <g id=\"axes_1\">\n","   <g id=\"patch_2\">\n","    <path d=\"M 52.160938 143.1 \n","L 331.160937 143.1 \n","L 331.160937 7.2 \n","L 52.160938 7.2 \n","z\n","\" style=\"fill: #ffffff\"/>\n","   </g>\n","   <g id=\"matplotlib.axis_1\">\n","    <g id=\"xtick_1\">\n","     <g id=\"line2d_1\">\n","      <path d=\"M 64.842756 143.1 \n","L 64.842756 7.2 \n","\" clip-path=\"url(#pbbc1dc489f)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n","     </g>\n","     <g id=\"line2d_2\">\n","      <defs>\n","       <path id=\"m7bc301e22e\" d=\"M 0 0 \n","L 0 3.5 \n","\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n","      </defs>\n","      <g>\n","       <use xlink:href=\"#m7bc301e22e\" x=\"64.842756\" y=\"143.1\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n","      </g>\n","     </g>\n","     <g id=\"text_1\">\n","      <!-- −8 -->\n","      <g transform=\"translate(57.471662 157.698438)scale(0.1 -0.1)\">\n","       <defs>\n","        <path id=\"DejaVuSans-2212\" d=\"M 678 2272 \n","L 4684 2272 \n","L 4684 1741 \n","L 678 1741 \n","L 678 2272 \n","z\n","\" transform=\"scale(0.015625)\"/>\n","        <path id=\"DejaVuSans-38\" d=\"M 2034 2216 \n","Q 1584 2216 1326 1975 \n","Q 1069 1734 1069 1313 \n","Q 1069 891 1326 650 \n","Q 1584 409 2034 409 \n","Q 2484 409 2743 651 \n","Q 3003 894 3003 1313 \n","Q 3003 1734 2745 1975 \n","Q 2488 2216 2034 2216 \n","z\n","M 1403 2484 \n","Q 997 2584 770 2862 \n","Q 544 3141 544 3541 \n","Q 544 4100 942 4425 \n","Q 1341 4750 2034 4750 \n","Q 2731 4750 3128 4425 \n","Q 3525 4100 3525 3541 \n","Q 3525 3141 3298 2862 \n","Q 3072 2584 2669 2484 \n","Q 3125 2378 3379 2068 \n","Q 3634 1759 3634 1313 \n","Q 3634 634 3220 271 \n","Q 2806 -91 2034 -91 \n","Q 1263 -91 848 271 \n","Q 434 634 434 1313 \n","Q 434 1759 690 2068 \n","Q 947 2378 1403 2484 \n","z\n","M 1172 3481 \n","Q 1172 3119 1398 2916 \n","Q 1625 2713 2034 2713 \n","Q 2441 2713 2670 2916 \n","Q 2900 3119 2900 3481 \n","Q 2900 3844 2670 4047 \n","Q 2441 4250 2034 4250 \n","Q 1625 4250 1398 4047 \n","Q 1172 3844 1172 3481 \n","z\n","\" transform=\"scale(0.015625)\"/>\n","       </defs>\n","       <use xlink:href=\"#DejaVuSans-2212\"/>\n","       <use xlink:href=\"#DejaVuSans-38\" x=\"83.789062\"/>\n","      </g>\n","     </g>\n","    </g>\n","    <g id=\"xtick_2\">\n","     <g id=\"line2d_3\">\n","      <path d=\"M 96.746702 143.1 \n","L 96.746702 7.2 \n","\" clip-path=\"url(#pbbc1dc489f)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n","     </g>\n","     <g id=\"line2d_4\">\n","      <g>\n","       <use xlink:href=\"#m7bc301e22e\" x=\"96.746702\" y=\"143.1\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n","      </g>\n","     </g>\n","     <g id=\"text_2\">\n","      <!-- −6 -->\n","      <g transform=\"translate(89.375608 157.698438)scale(0.1 -0.1)\">\n","       <defs>\n","        <path id=\"DejaVuSans-36\" d=\"M 2113 2584 \n","Q 1688 2584 1439 2293 \n","Q 1191 2003 1191 1497 \n","Q 1191 994 1439 701 \n","Q 1688 409 2113 409 \n","Q 2538 409 2786 701 \n","Q 3034 994 3034 1497 \n","Q 3034 2003 2786 2293 \n","Q 2538 2584 2113 2584 \n","z\n","M 3366 4563 \n","L 3366 3988 \n","Q 3128 4100 2886 4159 \n","Q 2644 4219 2406 4219 \n","Q 1781 4219 1451 3797 \n","Q 1122 3375 1075 2522 \n","Q 1259 2794 1537 2939 \n","Q 1816 3084 2150 3084 \n","Q 2853 3084 3261 2657 \n","Q 3669 2231 3669 1497 \n","Q 3669 778 3244 343 \n","Q 2819 -91 2113 -91 \n","Q 1303 -91 875 529 \n","Q 447 1150 447 2328 \n","Q 447 3434 972 4092 \n","Q 1497 4750 2381 4750 \n","Q 2619 4750 2861 4703 \n","Q 3103 4656 3366 4563 \n","z\n","\" transform=\"scale(0.015625)\"/>\n","       </defs>\n","       <use xlink:href=\"#DejaVuSans-2212\"/>\n","       <use xlink:href=\"#DejaVuSans-36\" x=\"83.789062\"/>\n","      </g>\n","     </g>\n","    </g>\n","    <g id=\"xtick_3\">\n","     <g id=\"line2d_5\">\n","      <path d=\"M 128.650647 143.1 \n","L 128.650647 7.2 \n","\" clip-path=\"url(#pbbc1dc489f)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n","     </g>\n","     <g id=\"line2d_6\">\n","      <g>\n","       <use xlink:href=\"#m7bc301e22e\" x=\"128.650647\" y=\"143.1\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n","      </g>\n","     </g>\n","     <g id=\"text_3\">\n","      <!-- −4 -->\n","      <g transform=\"translate(121.279554 157.698438)scale(0.1 -0.1)\">\n","       <defs>\n","        <path id=\"DejaVuSans-34\" d=\"M 2419 4116 \n","L 825 1625 \n","L 2419 1625 \n","L 2419 4116 \n","z\n","M 2253 4666 \n","L 3047 4666 \n","L 3047 1625 \n","L 3713 1625 \n","L 3713 1100 \n","L 3047 1100 \n","L 3047 0 \n","L 2419 0 \n","L 2419 1100 \n","L 313 1100 \n","L 313 1709 \n","L 2253 4666 \n","z\n","\" transform=\"scale(0.015625)\"/>\n","       </defs>\n","       <use xlink:href=\"#DejaVuSans-2212\"/>\n","       <use xlink:href=\"#DejaVuSans-34\" x=\"83.789062\"/>\n","      </g>\n","     </g>\n","    </g>\n","    <g id=\"xtick_4\">\n","     <g id=\"line2d_7\">\n","      <path d=\"M 160.554593 143.1 \n","L 160.554593 7.2 \n","\" clip-path=\"url(#pbbc1dc489f)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n","     </g>\n","     <g id=\"line2d_8\">\n","      <g>\n","       <use xlink:href=\"#m7bc301e22e\" x=\"160.554593\" y=\"143.1\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n","      </g>\n","     </g>\n","     <g id=\"text_4\">\n","      <!-- −2 -->\n","      <g transform=\"translate(153.1835 157.698438)scale(0.1 -0.1)\">\n","       <defs>\n","        <path id=\"DejaVuSans-32\" d=\"M 1228 531 \n","L 3431 531 \n","L 3431 0 \n","L 469 0 \n","L 469 531 \n","Q 828 903 1448 1529 \n","Q 2069 2156 2228 2338 \n","Q 2531 2678 2651 2914 \n","Q 2772 3150 2772 3378 \n","Q 2772 3750 2511 3984 \n","Q 2250 4219 1831 4219 \n","Q 1534 4219 1204 4116 \n","Q 875 4013 500 3803 \n","L 500 4441 \n","Q 881 4594 1212 4672 \n","Q 1544 4750 1819 4750 \n","Q 2544 4750 2975 4387 \n","Q 3406 4025 3406 3419 \n","Q 3406 3131 3298 2873 \n","Q 3191 2616 2906 2266 \n","Q 2828 2175 2409 1742 \n","Q 1991 1309 1228 531 \n","z\n","\" transform=\"scale(0.015625)\"/>\n","       </defs>\n","       <use xlink:href=\"#DejaVuSans-2212\"/>\n","       <use xlink:href=\"#DejaVuSans-32\" x=\"83.789062\"/>\n","      </g>\n","     </g>\n","    </g>\n","    <g id=\"xtick_5\">\n","     <g id=\"line2d_9\">\n","      <path d=\"M 192.458539 143.1 \n","L 192.458539 7.2 \n","\" clip-path=\"url(#pbbc1dc489f)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n","     </g>\n","     <g id=\"line2d_10\">\n","      <g>\n","       <use xlink:href=\"#m7bc301e22e\" x=\"192.458539\" y=\"143.1\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n","      </g>\n","     </g>\n","     <g id=\"text_5\">\n","      <!-- 0 -->\n","      <g transform=\"translate(189.277289 157.698438)scale(0.1 -0.1)\">\n","       <defs>\n","        <path id=\"DejaVuSans-30\" d=\"M 2034 4250 \n","Q 1547 4250 1301 3770 \n","Q 1056 3291 1056 2328 \n","Q 1056 1369 1301 889 \n","Q 1547 409 2034 409 \n","Q 2525 409 2770 889 \n","Q 3016 1369 3016 2328 \n","Q 3016 3291 2770 3770 \n","Q 2525 4250 2034 4250 \n","z\n","M 2034 4750 \n","Q 2819 4750 3233 4129 \n","Q 3647 3509 3647 2328 \n","Q 3647 1150 3233 529 \n","Q 2819 -91 2034 -91 \n","Q 1250 -91 836 529 \n","Q 422 1150 422 2328 \n","Q 422 3509 836 4129 \n","Q 1250 4750 2034 4750 \n","z\n","\" transform=\"scale(0.015625)\"/>\n","       </defs>\n","       <use xlink:href=\"#DejaVuSans-30\"/>\n","      </g>\n","     </g>\n","    </g>\n","    <g id=\"xtick_6\">\n","     <g id=\"line2d_11\">\n","      <path d=\"M 224.362485 143.1 \n","L 224.362485 7.2 \n","\" clip-path=\"url(#pbbc1dc489f)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n","     </g>\n","     <g id=\"line2d_12\">\n","      <g>\n","       <use xlink:href=\"#m7bc301e22e\" x=\"224.362485\" y=\"143.1\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n","      </g>\n","     </g>\n","     <g id=\"text_6\">\n","      <!-- 2 -->\n","      <g transform=\"translate(221.181235 157.698438)scale(0.1 -0.1)\">\n","       <use xlink:href=\"#DejaVuSans-32\"/>\n","      </g>\n","     </g>\n","    </g>\n","    <g id=\"xtick_7\">\n","     <g id=\"line2d_13\">\n","      <path d=\"M 256.266431 143.1 \n","L 256.266431 7.2 \n","\" clip-path=\"url(#pbbc1dc489f)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n","     </g>\n","     <g id=\"line2d_14\">\n","      <g>\n","       <use xlink:href=\"#m7bc301e22e\" x=\"256.266431\" y=\"143.1\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n","      </g>\n","     </g>\n","     <g id=\"text_7\">\n","      <!-- 4 -->\n","      <g transform=\"translate(253.085181 157.698438)scale(0.1 -0.1)\">\n","       <use xlink:href=\"#DejaVuSans-34\"/>\n","      </g>\n","     </g>\n","    </g>\n","    <g id=\"xtick_8\">\n","     <g id=\"line2d_15\">\n","      <path d=\"M 288.170377 143.1 \n","L 288.170377 7.2 \n","\" clip-path=\"url(#pbbc1dc489f)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n","     </g>\n","     <g id=\"line2d_16\">\n","      <g>\n","       <use xlink:href=\"#m7bc301e22e\" x=\"288.170377\" y=\"143.1\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n","      </g>\n","     </g>\n","     <g id=\"text_8\">\n","      <!-- 6 -->\n","      <g transform=\"translate(284.989127 157.698438)scale(0.1 -0.1)\">\n","       <use xlink:href=\"#DejaVuSans-36\"/>\n","      </g>\n","     </g>\n","    </g>\n","    <g id=\"xtick_9\">\n","     <g id=\"line2d_17\">\n","      <path d=\"M 320.074323 143.1 \n","L 320.074323 7.2 \n","\" clip-path=\"url(#pbbc1dc489f)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n","     </g>\n","     <g id=\"line2d_18\">\n","      <g>\n","       <use xlink:href=\"#m7bc301e22e\" x=\"320.074323\" y=\"143.1\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n","      </g>\n","     </g>\n","     <g id=\"text_9\">\n","      <!-- 8 -->\n","      <g transform=\"translate(316.893073 157.698438)scale(0.1 -0.1)\">\n","       <use xlink:href=\"#DejaVuSans-38\"/>\n","      </g>\n","     </g>\n","    </g>\n","    <g id=\"text_10\">\n","     <!-- x -->\n","     <g transform=\"translate(188.701562 171.376563)scale(0.1 -0.1)\">\n","      <defs>\n","       <path id=\"DejaVuSans-78\" d=\"M 3513 3500 \n","L 2247 1797 \n","L 3578 0 \n","L 2900 0 \n","L 1881 1375 \n","L 863 0 \n","L 184 0 \n","L 1544 1831 \n","L 300 3500 \n","L 978 3500 \n","L 1906 2253 \n","L 2834 3500 \n","L 3513 3500 \n","z\n","\" transform=\"scale(0.015625)\"/>\n","      </defs>\n","      <use xlink:href=\"#DejaVuSans-78\"/>\n","     </g>\n","    </g>\n","   </g>\n","   <g id=\"matplotlib.axis_2\">\n","    <g id=\"ytick_1\">\n","     <g id=\"line2d_19\">\n","      <path d=\"M 52.160938 136.922742 \n","L 331.160937 136.922742 \n","\" clip-path=\"url(#pbbc1dc489f)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n","     </g>\n","     <g id=\"line2d_20\">\n","      <defs>\n","       <path id=\"ma62e7b202c\" d=\"M 0 0 \n","L -3.5 0 \n","\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n","      </defs>\n","      <g>\n","       <use xlink:href=\"#ma62e7b202c\" x=\"52.160938\" y=\"136.922742\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n","      </g>\n","     </g>\n","     <g id=\"text_11\">\n","      <!-- −1.0 -->\n","      <g transform=\"translate(20.878125 140.721961)scale(0.1 -0.1)\">\n","       <defs>\n","        <path id=\"DejaVuSans-31\" d=\"M 794 531 \n","L 1825 531 \n","L 1825 4091 \n","L 703 3866 \n","L 703 4441 \n","L 1819 4666 \n","L 2450 4666 \n","L 2450 531 \n","L 3481 531 \n","L 3481 0 \n","L 794 0 \n","L 794 531 \n","z\n","\" transform=\"scale(0.015625)\"/>\n","        <path id=\"DejaVuSans-2e\" d=\"M 684 794 \n","L 1344 794 \n","L 1344 0 \n","L 684 0 \n","L 684 794 \n","z\n","\" transform=\"scale(0.015625)\"/>\n","       </defs>\n","       <use xlink:href=\"#DejaVuSans-2212\"/>\n","       <use xlink:href=\"#DejaVuSans-31\" x=\"83.789062\"/>\n","       <use xlink:href=\"#DejaVuSans-2e\" x=\"147.412109\"/>\n","       <use xlink:href=\"#DejaVuSans-30\" x=\"179.199219\"/>\n","      </g>\n","     </g>\n","    </g>\n","    <g id=\"ytick_2\">\n","     <g id=\"line2d_21\">\n","      <path d=\"M 52.160938 106.03637 \n","L 331.160937 106.03637 \n","\" clip-path=\"url(#pbbc1dc489f)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n","     </g>\n","     <g id=\"line2d_22\">\n","      <g>\n","       <use xlink:href=\"#ma62e7b202c\" x=\"52.160938\" y=\"106.03637\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n","      </g>\n","     </g>\n","     <g id=\"text_12\">\n","      <!-- −0.5 -->\n","      <g transform=\"translate(20.878125 109.835589)scale(0.1 -0.1)\">\n","       <defs>\n","        <path id=\"DejaVuSans-35\" d=\"M 691 4666 \n","L 3169 4666 \n","L 3169 4134 \n","L 1269 4134 \n","L 1269 2991 \n","Q 1406 3038 1543 3061 \n","Q 1681 3084 1819 3084 \n","Q 2600 3084 3056 2656 \n","Q 3513 2228 3513 1497 \n","Q 3513 744 3044 326 \n","Q 2575 -91 1722 -91 \n","Q 1428 -91 1123 -41 \n","Q 819 9 494 109 \n","L 494 744 \n","Q 775 591 1075 516 \n","Q 1375 441 1709 441 \n","Q 2250 441 2565 725 \n","Q 2881 1009 2881 1497 \n","Q 2881 1984 2565 2268 \n","Q 2250 2553 1709 2553 \n","Q 1456 2553 1204 2497 \n","Q 953 2441 691 2322 \n","L 691 4666 \n","z\n","\" transform=\"scale(0.015625)\"/>\n","       </defs>\n","       <use xlink:href=\"#DejaVuSans-2212\"/>\n","       <use xlink:href=\"#DejaVuSans-30\" x=\"83.789062\"/>\n","       <use xlink:href=\"#DejaVuSans-2e\" x=\"147.412109\"/>\n","       <use xlink:href=\"#DejaVuSans-35\" x=\"179.199219\"/>\n","      </g>\n","     </g>\n","    </g>\n","    <g id=\"ytick_3\">\n","     <g id=\"line2d_23\">\n","      <path d=\"M 52.160938 75.149998 \n","L 331.160937 75.149998 \n","\" clip-path=\"url(#pbbc1dc489f)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n","     </g>\n","     <g id=\"line2d_24\">\n","      <g>\n","       <use xlink:href=\"#ma62e7b202c\" x=\"52.160938\" y=\"75.149998\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n","      </g>\n","     </g>\n","     <g id=\"text_13\">\n","      <!-- 0.0 -->\n","      <g transform=\"translate(29.257812 78.949217)scale(0.1 -0.1)\">\n","       <use xlink:href=\"#DejaVuSans-30\"/>\n","       <use xlink:href=\"#DejaVuSans-2e\" x=\"63.623047\"/>\n","       <use xlink:href=\"#DejaVuSans-30\" x=\"95.410156\"/>\n","      </g>\n","     </g>\n","    </g>\n","    <g id=\"ytick_4\">\n","     <g id=\"line2d_25\">\n","      <path d=\"M 52.160938 44.263626 \n","L 331.160937 44.263626 \n","\" clip-path=\"url(#pbbc1dc489f)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n","     </g>\n","     <g id=\"line2d_26\">\n","      <g>\n","       <use xlink:href=\"#ma62e7b202c\" x=\"52.160938\" y=\"44.263626\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n","      </g>\n","     </g>\n","     <g id=\"text_14\">\n","      <!-- 0.5 -->\n","      <g transform=\"translate(29.257812 48.062845)scale(0.1 -0.1)\">\n","       <use xlink:href=\"#DejaVuSans-30\"/>\n","       <use xlink:href=\"#DejaVuSans-2e\" x=\"63.623047\"/>\n","       <use xlink:href=\"#DejaVuSans-35\" x=\"95.410156\"/>\n","      </g>\n","     </g>\n","    </g>\n","    <g id=\"ytick_5\">\n","     <g id=\"line2d_27\">\n","      <path d=\"M 52.160938 13.377254 \n","L 331.160937 13.377254 \n","\" clip-path=\"url(#pbbc1dc489f)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n","     </g>\n","     <g id=\"line2d_28\">\n","      <g>\n","       <use xlink:href=\"#ma62e7b202c\" x=\"52.160938\" y=\"13.377254\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n","      </g>\n","     </g>\n","     <g id=\"text_15\">\n","      <!-- 1.0 -->\n","      <g transform=\"translate(29.257812 17.176473)scale(0.1 -0.1)\">\n","       <use xlink:href=\"#DejaVuSans-31\"/>\n","       <use xlink:href=\"#DejaVuSans-2e\" x=\"63.623047\"/>\n","       <use xlink:href=\"#DejaVuSans-30\" x=\"95.410156\"/>\n","      </g>\n","     </g>\n","    </g>\n","    <g id=\"text_16\">\n","     <!-- tanh(x) -->\n","     <g transform=\"translate(14.798437 93.372656)rotate(-90)scale(0.1 -0.1)\">\n","      <defs>\n","       <path id=\"DejaVuSans-74\" d=\"M 1172 4494 \n","L 1172 3500 \n","L 2356 3500 \n","L 2356 3053 \n","L 1172 3053 \n","L 1172 1153 \n","Q 1172 725 1289 603 \n","Q 1406 481 1766 481 \n","L 2356 481 \n","L 2356 0 \n","L 1766 0 \n","Q 1100 0 847 248 \n","Q 594 497 594 1153 \n","L 594 3053 \n","L 172 3053 \n","L 172 3500 \n","L 594 3500 \n","L 594 4494 \n","L 1172 4494 \n","z\n","\" transform=\"scale(0.015625)\"/>\n","       <path id=\"DejaVuSans-61\" d=\"M 2194 1759 \n","Q 1497 1759 1228 1600 \n","Q 959 1441 959 1056 \n","Q 959 750 1161 570 \n","Q 1363 391 1709 391 \n","Q 2188 391 2477 730 \n","Q 2766 1069 2766 1631 \n","L 2766 1759 \n","L 2194 1759 \n","z\n","M 3341 1997 \n","L 3341 0 \n","L 2766 0 \n","L 2766 531 \n","Q 2569 213 2275 61 \n","Q 1981 -91 1556 -91 \n","Q 1019 -91 701 211 \n","Q 384 513 384 1019 \n","Q 384 1609 779 1909 \n","Q 1175 2209 1959 2209 \n","L 2766 2209 \n","L 2766 2266 \n","Q 2766 2663 2505 2880 \n","Q 2244 3097 1772 3097 \n","Q 1472 3097 1187 3025 \n","Q 903 2953 641 2809 \n","L 641 3341 \n","Q 956 3463 1253 3523 \n","Q 1550 3584 1831 3584 \n","Q 2591 3584 2966 3190 \n","Q 3341 2797 3341 1997 \n","z\n","\" transform=\"scale(0.015625)\"/>\n","       <path id=\"DejaVuSans-6e\" d=\"M 3513 2113 \n","L 3513 0 \n","L 2938 0 \n","L 2938 2094 \n","Q 2938 2591 2744 2837 \n","Q 2550 3084 2163 3084 \n","Q 1697 3084 1428 2787 \n","Q 1159 2491 1159 1978 \n","L 1159 0 \n","L 581 0 \n","L 581 3500 \n","L 1159 3500 \n","L 1159 2956 \n","Q 1366 3272 1645 3428 \n","Q 1925 3584 2291 3584 \n","Q 2894 3584 3203 3211 \n","Q 3513 2838 3513 2113 \n","z\n","\" transform=\"scale(0.015625)\"/>\n","       <path id=\"DejaVuSans-68\" d=\"M 3513 2113 \n","L 3513 0 \n","L 2938 0 \n","L 2938 2094 \n","Q 2938 2591 2744 2837 \n","Q 2550 3084 2163 3084 \n","Q 1697 3084 1428 2787 \n","Q 1159 2491 1159 1978 \n","L 1159 0 \n","L 581 0 \n","L 581 4863 \n","L 1159 4863 \n","L 1159 2956 \n","Q 1366 3272 1645 3428 \n","Q 1925 3584 2291 3584 \n","Q 2894 3584 3203 3211 \n","Q 3513 2838 3513 2113 \n","z\n","\" transform=\"scale(0.015625)\"/>\n","       <path id=\"DejaVuSans-28\" d=\"M 1984 4856 \n","Q 1566 4138 1362 3434 \n","Q 1159 2731 1159 2009 \n","Q 1159 1288 1364 580 \n","Q 1569 -128 1984 -844 \n","L 1484 -844 \n","Q 1016 -109 783 600 \n","Q 550 1309 550 2009 \n","Q 550 2706 781 3412 \n","Q 1013 4119 1484 4856 \n","L 1984 4856 \n","z\n","\" transform=\"scale(0.015625)\"/>\n","       <path id=\"DejaVuSans-29\" d=\"M 513 4856 \n","L 1013 4856 \n","Q 1481 4119 1714 3412 \n","Q 1947 2706 1947 2009 \n","Q 1947 1309 1714 600 \n","Q 1481 -109 1013 -844 \n","L 513 -844 \n","Q 928 -128 1133 580 \n","Q 1338 1288 1338 2009 \n","Q 1338 2731 1133 3434 \n","Q 928 4138 513 4856 \n","z\n","\" transform=\"scale(0.015625)\"/>\n","      </defs>\n","      <use xlink:href=\"#DejaVuSans-74\"/>\n","      <use xlink:href=\"#DejaVuSans-61\" x=\"39.208984\"/>\n","      <use xlink:href=\"#DejaVuSans-6e\" x=\"100.488281\"/>\n","      <use xlink:href=\"#DejaVuSans-68\" x=\"163.867188\"/>\n","      <use xlink:href=\"#DejaVuSans-28\" x=\"227.246094\"/>\n","      <use xlink:href=\"#DejaVuSans-78\" x=\"266.259766\"/>\n","      <use xlink:href=\"#DejaVuSans-29\" x=\"325.439453\"/>\n","     </g>\n","    </g>\n","   </g>\n","   <g id=\"line2d_29\">\n","    <path d=\"M 64.842756 136.922727 \n","L 135.031438 136.830572 \n","L 144.60262 136.617262 \n","L 150.983407 136.244933 \n","L 155.768998 135.693242 \n","L 158.959394 135.097478 \n","L 162.149789 134.219412 \n","L 163.744987 133.636804 \n","L 165.340184 132.932783 \n","L 166.935382 132.083992 \n","L 168.53058 131.06349 \n","L 170.125777 129.840599 \n","L 171.720973 128.381004 \n","L 173.316171 126.647133 \n","L 174.911369 124.599019 \n","L 176.506566 122.19576 \n","L 178.101763 119.397683 \n","L 179.696961 116.16937 \n","L 181.292158 112.483455 \n","L 182.887355 108.325025 \n","L 184.482553 103.696243 \n","L 186.07775 98.620489 \n","L 189.268144 87.342415 \n","L 197.244131 57.154817 \n","L 198.839328 51.679508 \n","L 200.434526 46.603753 \n","L 202.029723 41.974971 \n","L 203.62492 37.816541 \n","L 205.220118 34.130626 \n","L 206.815315 30.902314 \n","L 208.410512 28.104236 \n","L 210.00571 25.700977 \n","L 211.600907 23.652864 \n","L 213.196103 21.918996 \n","L 214.791301 20.459397 \n","L 216.386499 19.236506 \n","L 217.981696 18.216004 \n","L 219.576894 17.367213 \n","L 222.767287 16.080588 \n","L 225.957681 15.202518 \n","L 229.148076 14.606754 \n","L 233.933671 14.055063 \n","L 238.719262 13.750165 \n","L 246.695249 13.514705 \n","L 261.052026 13.399994 \n","L 312.098336 13.377291 \n","L 318.479119 13.377273 \n","L 318.479119 13.377273 \n","\" clip-path=\"url(#pbbc1dc489f)\" style=\"fill: none; stroke: #1f77b4; stroke-width: 1.5; stroke-linecap: square\"/>\n","   </g>\n","   <g id=\"patch_3\">\n","    <path d=\"M 52.160938 143.1 \n","L 52.160938 7.2 \n","\" style=\"fill: none; stroke: #000000; stroke-width: 0.8; stroke-linejoin: miter; stroke-linecap: square\"/>\n","   </g>\n","   <g id=\"patch_4\">\n","    <path d=\"M 331.160937 143.1 \n","L 331.160937 7.2 \n","\" style=\"fill: none; stroke: #000000; stroke-width: 0.8; stroke-linejoin: miter; stroke-linecap: square\"/>\n","   </g>\n","   <g id=\"patch_5\">\n","    <path d=\"M 52.160938 143.1 \n","L 331.160938 143.1 \n","\" style=\"fill: none; stroke: #000000; stroke-width: 0.8; stroke-linejoin: miter; stroke-linecap: square\"/>\n","   </g>\n","   <g id=\"patch_6\">\n","    <path d=\"M 52.160938 7.2 \n","L 331.160938 7.2 \n","\" style=\"fill: none; stroke: #000000; stroke-width: 0.8; stroke-linejoin: miter; stroke-linecap: square\"/>\n","   </g>\n","  </g>\n"," </g>\n"," <defs>\n","  <clipPath id=\"pbbc1dc489f\">\n","   <rect x=\"52.160938\" y=\"7.2\" width=\"279\" height=\"135.9\"/>\n","  </clipPath>\n"," </defs>\n","</svg>\n"],"text/plain":["<Figure size 360x180 with 1 Axes>"]},"metadata":{"needs_background":"light"},"output_type":"display_data"}],"source":["y = torch.tanh(x)\n","d2l.plot(x.detach(), y.detach(), 'x', 'tanh(x)', figsize=(5, 2.5))"]},{"cell_type":"markdown","id":"45e1c07d","metadata":{"origin_pos":24,"id":"45e1c07d"},"source":["tanh函数的导数是：\n","\n","$$\\frac{d}{dx} \\operatorname{tanh}(x) = 1 - \\operatorname{tanh}^2(x).$$\n","\n","tanh函数的导数图像如下所示。\n","当输入接近0时，tanh函数的导数接近最大值1。\n","与我们在sigmoid函数图像中看到的类似，\n","输入在任一方向上越远离0点，导数越接近0。\n"]},{"cell_type":"code","execution_count":null,"id":"cf94d742","metadata":{"execution":{"iopub.execute_input":"2022-07-31T22:40:54.990036Z","iopub.status.busy":"2022-07-31T22:40:54.989433Z","iopub.status.idle":"2022-07-31T22:40:55.187873Z","shell.execute_reply":"2022-07-31T22:40:55.187240Z"},"origin_pos":26,"tab":["pytorch"],"id":"cf94d742","outputId":"6378dba3-ecf3-43d3-d29e-0d586095275a"},"outputs":[{"data":{"image/svg+xml":["<?xml version=\"1.0\" encoding=\"utf-8\" standalone=\"no\"?>\n","<!DOCTYPE svg PUBLIC \"-//W3C//DTD SVG 1.1//EN\"\n","  \"http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd\">\n","<svg xmlns:xlink=\"http://www.w3.org/1999/xlink\" width=\"329.98125pt\" height=\"180.65625pt\" viewBox=\"0 0 329.98125 180.65625\" xmlns=\"http://www.w3.org/2000/svg\" version=\"1.1\">\n"," <metadata>\n","  <rdf:RDF xmlns:dc=\"http://purl.org/dc/elements/1.1/\" xmlns:cc=\"http://creativecommons.org/ns#\" xmlns:rdf=\"http://www.w3.org/1999/02/22-rdf-syntax-ns#\">\n","   <cc:Work>\n","    <dc:type rdf:resource=\"http://purl.org/dc/dcmitype/StillImage\"/>\n","    <dc:date>2022-07-31T22:40:55.144215</dc:date>\n","    <dc:format>image/svg+xml</dc:format>\n","    <dc:creator>\n","     <cc:Agent>\n","      <dc:title>Matplotlib v3.5.1, https://matplotlib.org/</dc:title>\n","     </cc:Agent>\n","    </dc:creator>\n","   </cc:Work>\n","  </rdf:RDF>\n"," </metadata>\n"," <defs>\n","  <style type=\"text/css\">*{stroke-linejoin: round; stroke-linecap: butt}</style>\n"," </defs>\n"," <g id=\"figure_1\">\n","  <g id=\"patch_1\">\n","   <path d=\"M 0 180.65625 \n","L 329.98125 180.65625 \n","L 329.98125 0 \n","L 0 0 \n","L 0 180.65625 \n","z\n","\" style=\"fill: none\"/>\n","  </g>\n","  <g id=\"axes_1\">\n","   <g id=\"patch_2\">\n","    <path d=\"M 43.78125 143.1 \n","L 322.78125 143.1 \n","L 322.78125 7.2 \n","L 43.78125 7.2 \n","z\n","\" style=\"fill: #ffffff\"/>\n","   </g>\n","   <g id=\"matplotlib.axis_1\">\n","    <g id=\"xtick_1\">\n","     <g id=\"line2d_1\">\n","      <path d=\"M 56.463068 143.1 \n","L 56.463068 7.2 \n","\" clip-path=\"url(#p5bd91fec43)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n","     </g>\n","     <g id=\"line2d_2\">\n","      <defs>\n","       <path id=\"m3abe2df531\" d=\"M 0 0 \n","L 0 3.5 \n","\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n","      </defs>\n","      <g>\n","       <use xlink:href=\"#m3abe2df531\" x=\"56.463068\" y=\"143.1\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n","      </g>\n","     </g>\n","     <g id=\"text_1\">\n","      <!-- −8 -->\n","      <g transform=\"translate(49.091974 157.698438)scale(0.1 -0.1)\">\n","       <defs>\n","        <path id=\"DejaVuSans-2212\" d=\"M 678 2272 \n","L 4684 2272 \n","L 4684 1741 \n","L 678 1741 \n","L 678 2272 \n","z\n","\" transform=\"scale(0.015625)\"/>\n","        <path id=\"DejaVuSans-38\" d=\"M 2034 2216 \n","Q 1584 2216 1326 1975 \n","Q 1069 1734 1069 1313 \n","Q 1069 891 1326 650 \n","Q 1584 409 2034 409 \n","Q 2484 409 2743 651 \n","Q 3003 894 3003 1313 \n","Q 3003 1734 2745 1975 \n","Q 2488 2216 2034 2216 \n","z\n","M 1403 2484 \n","Q 997 2584 770 2862 \n","Q 544 3141 544 3541 \n","Q 544 4100 942 4425 \n","Q 1341 4750 2034 4750 \n","Q 2731 4750 3128 4425 \n","Q 3525 4100 3525 3541 \n","Q 3525 3141 3298 2862 \n","Q 3072 2584 2669 2484 \n","Q 3125 2378 3379 2068 \n","Q 3634 1759 3634 1313 \n","Q 3634 634 3220 271 \n","Q 2806 -91 2034 -91 \n","Q 1263 -91 848 271 \n","Q 434 634 434 1313 \n","Q 434 1759 690 2068 \n","Q 947 2378 1403 2484 \n","z\n","M 1172 3481 \n","Q 1172 3119 1398 2916 \n","Q 1625 2713 2034 2713 \n","Q 2441 2713 2670 2916 \n","Q 2900 3119 2900 3481 \n","Q 2900 3844 2670 4047 \n","Q 2441 4250 2034 4250 \n","Q 1625 4250 1398 4047 \n","Q 1172 3844 1172 3481 \n","z\n","\" transform=\"scale(0.015625)\"/>\n","       </defs>\n","       <use xlink:href=\"#DejaVuSans-2212\"/>\n","       <use xlink:href=\"#DejaVuSans-38\" x=\"83.789062\"/>\n","      </g>\n","     </g>\n","    </g>\n","    <g id=\"xtick_2\">\n","     <g id=\"line2d_3\">\n","      <path d=\"M 88.367014 143.1 \n","L 88.367014 7.2 \n","\" clip-path=\"url(#p5bd91fec43)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n","     </g>\n","     <g id=\"line2d_4\">\n","      <g>\n","       <use xlink:href=\"#m3abe2df531\" x=\"88.367014\" y=\"143.1\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n","      </g>\n","     </g>\n","     <g id=\"text_2\">\n","      <!-- −6 -->\n","      <g transform=\"translate(80.99592 157.698438)scale(0.1 -0.1)\">\n","       <defs>\n","        <path id=\"DejaVuSans-36\" d=\"M 2113 2584 \n","Q 1688 2584 1439 2293 \n","Q 1191 2003 1191 1497 \n","Q 1191 994 1439 701 \n","Q 1688 409 2113 409 \n","Q 2538 409 2786 701 \n","Q 3034 994 3034 1497 \n","Q 3034 2003 2786 2293 \n","Q 2538 2584 2113 2584 \n","z\n","M 3366 4563 \n","L 3366 3988 \n","Q 3128 4100 2886 4159 \n","Q 2644 4219 2406 4219 \n","Q 1781 4219 1451 3797 \n","Q 1122 3375 1075 2522 \n","Q 1259 2794 1537 2939 \n","Q 1816 3084 2150 3084 \n","Q 2853 3084 3261 2657 \n","Q 3669 2231 3669 1497 \n","Q 3669 778 3244 343 \n","Q 2819 -91 2113 -91 \n","Q 1303 -91 875 529 \n","Q 447 1150 447 2328 \n","Q 447 3434 972 4092 \n","Q 1497 4750 2381 4750 \n","Q 2619 4750 2861 4703 \n","Q 3103 4656 3366 4563 \n","z\n","\" transform=\"scale(0.015625)\"/>\n","       </defs>\n","       <use xlink:href=\"#DejaVuSans-2212\"/>\n","       <use xlink:href=\"#DejaVuSans-36\" x=\"83.789062\"/>\n","      </g>\n","     </g>\n","    </g>\n","    <g id=\"xtick_3\">\n","     <g id=\"line2d_5\">\n","      <path d=\"M 120.27096 143.1 \n","L 120.27096 7.2 \n","\" clip-path=\"url(#p5bd91fec43)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n","     </g>\n","     <g id=\"line2d_6\">\n","      <g>\n","       <use xlink:href=\"#m3abe2df531\" x=\"120.27096\" y=\"143.1\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n","      </g>\n","     </g>\n","     <g id=\"text_3\">\n","      <!-- −4 -->\n","      <g transform=\"translate(112.899866 157.698438)scale(0.1 -0.1)\">\n","       <defs>\n","        <path id=\"DejaVuSans-34\" d=\"M 2419 4116 \n","L 825 1625 \n","L 2419 1625 \n","L 2419 4116 \n","z\n","M 2253 4666 \n","L 3047 4666 \n","L 3047 1625 \n","L 3713 1625 \n","L 3713 1100 \n","L 3047 1100 \n","L 3047 0 \n","L 2419 0 \n","L 2419 1100 \n","L 313 1100 \n","L 313 1709 \n","L 2253 4666 \n","z\n","\" transform=\"scale(0.015625)\"/>\n","       </defs>\n","       <use xlink:href=\"#DejaVuSans-2212\"/>\n","       <use xlink:href=\"#DejaVuSans-34\" x=\"83.789062\"/>\n","      </g>\n","     </g>\n","    </g>\n","    <g id=\"xtick_4\">\n","     <g id=\"line2d_7\">\n","      <path d=\"M 152.174906 143.1 \n","L 152.174906 7.2 \n","\" clip-path=\"url(#p5bd91fec43)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n","     </g>\n","     <g id=\"line2d_8\">\n","      <g>\n","       <use xlink:href=\"#m3abe2df531\" x=\"152.174906\" y=\"143.1\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n","      </g>\n","     </g>\n","     <g id=\"text_4\">\n","      <!-- −2 -->\n","      <g transform=\"translate(144.803812 157.698438)scale(0.1 -0.1)\">\n","       <defs>\n","        <path id=\"DejaVuSans-32\" d=\"M 1228 531 \n","L 3431 531 \n","L 3431 0 \n","L 469 0 \n","L 469 531 \n","Q 828 903 1448 1529 \n","Q 2069 2156 2228 2338 \n","Q 2531 2678 2651 2914 \n","Q 2772 3150 2772 3378 \n","Q 2772 3750 2511 3984 \n","Q 2250 4219 1831 4219 \n","Q 1534 4219 1204 4116 \n","Q 875 4013 500 3803 \n","L 500 4441 \n","Q 881 4594 1212 4672 \n","Q 1544 4750 1819 4750 \n","Q 2544 4750 2975 4387 \n","Q 3406 4025 3406 3419 \n","Q 3406 3131 3298 2873 \n","Q 3191 2616 2906 2266 \n","Q 2828 2175 2409 1742 \n","Q 1991 1309 1228 531 \n","z\n","\" transform=\"scale(0.015625)\"/>\n","       </defs>\n","       <use xlink:href=\"#DejaVuSans-2212\"/>\n","       <use xlink:href=\"#DejaVuSans-32\" x=\"83.789062\"/>\n","      </g>\n","     </g>\n","    </g>\n","    <g id=\"xtick_5\">\n","     <g id=\"line2d_9\">\n","      <path d=\"M 184.078852 143.1 \n","L 184.078852 7.2 \n","\" clip-path=\"url(#p5bd91fec43)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n","     </g>\n","     <g id=\"line2d_10\">\n","      <g>\n","       <use xlink:href=\"#m3abe2df531\" x=\"184.078852\" y=\"143.1\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n","      </g>\n","     </g>\n","     <g id=\"text_5\">\n","      <!-- 0 -->\n","      <g transform=\"translate(180.897602 157.698438)scale(0.1 -0.1)\">\n","       <defs>\n","        <path id=\"DejaVuSans-30\" d=\"M 2034 4250 \n","Q 1547 4250 1301 3770 \n","Q 1056 3291 1056 2328 \n","Q 1056 1369 1301 889 \n","Q 1547 409 2034 409 \n","Q 2525 409 2770 889 \n","Q 3016 1369 3016 2328 \n","Q 3016 3291 2770 3770 \n","Q 2525 4250 2034 4250 \n","z\n","M 2034 4750 \n","Q 2819 4750 3233 4129 \n","Q 3647 3509 3647 2328 \n","Q 3647 1150 3233 529 \n","Q 2819 -91 2034 -91 \n","Q 1250 -91 836 529 \n","Q 422 1150 422 2328 \n","Q 422 3509 836 4129 \n","Q 1250 4750 2034 4750 \n","z\n","\" transform=\"scale(0.015625)\"/>\n","       </defs>\n","       <use xlink:href=\"#DejaVuSans-30\"/>\n","      </g>\n","     </g>\n","    </g>\n","    <g id=\"xtick_6\">\n","     <g id=\"line2d_11\">\n","      <path d=\"M 215.982798 143.1 \n","L 215.982798 7.2 \n","\" clip-path=\"url(#p5bd91fec43)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n","     </g>\n","     <g id=\"line2d_12\">\n","      <g>\n","       <use xlink:href=\"#m3abe2df531\" x=\"215.982798\" y=\"143.1\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n","      </g>\n","     </g>\n","     <g id=\"text_6\">\n","      <!-- 2 -->\n","      <g transform=\"translate(212.801548 157.698438)scale(0.1 -0.1)\">\n","       <use xlink:href=\"#DejaVuSans-32\"/>\n","      </g>\n","     </g>\n","    </g>\n","    <g id=\"xtick_7\">\n","     <g id=\"line2d_13\">\n","      <path d=\"M 247.886743 143.1 \n","L 247.886743 7.2 \n","\" clip-path=\"url(#p5bd91fec43)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n","     </g>\n","     <g id=\"line2d_14\">\n","      <g>\n","       <use xlink:href=\"#m3abe2df531\" x=\"247.886743\" y=\"143.1\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n","      </g>\n","     </g>\n","     <g id=\"text_7\">\n","      <!-- 4 -->\n","      <g transform=\"translate(244.705493 157.698438)scale(0.1 -0.1)\">\n","       <use xlink:href=\"#DejaVuSans-34\"/>\n","      </g>\n","     </g>\n","    </g>\n","    <g id=\"xtick_8\">\n","     <g id=\"line2d_15\">\n","      <path d=\"M 279.790689 143.1 \n","L 279.790689 7.2 \n","\" clip-path=\"url(#p5bd91fec43)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n","     </g>\n","     <g id=\"line2d_16\">\n","      <g>\n","       <use xlink:href=\"#m3abe2df531\" x=\"279.790689\" y=\"143.1\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n","      </g>\n","     </g>\n","     <g id=\"text_8\">\n","      <!-- 6 -->\n","      <g transform=\"translate(276.609439 157.698438)scale(0.1 -0.1)\">\n","       <use xlink:href=\"#DejaVuSans-36\"/>\n","      </g>\n","     </g>\n","    </g>\n","    <g id=\"xtick_9\">\n","     <g id=\"line2d_17\">\n","      <path d=\"M 311.694635 143.1 \n","L 311.694635 7.2 \n","\" clip-path=\"url(#p5bd91fec43)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n","     </g>\n","     <g id=\"line2d_18\">\n","      <g>\n","       <use xlink:href=\"#m3abe2df531\" x=\"311.694635\" y=\"143.1\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n","      </g>\n","     </g>\n","     <g id=\"text_9\">\n","      <!-- 8 -->\n","      <g transform=\"translate(308.513385 157.698438)scale(0.1 -0.1)\">\n","       <use xlink:href=\"#DejaVuSans-38\"/>\n","      </g>\n","     </g>\n","    </g>\n","    <g id=\"text_10\">\n","     <!-- x -->\n","     <g transform=\"translate(180.321875 171.376563)scale(0.1 -0.1)\">\n","      <defs>\n","       <path id=\"DejaVuSans-78\" d=\"M 3513 3500 \n","L 2247 1797 \n","L 3578 0 \n","L 2900 0 \n","L 1881 1375 \n","L 863 0 \n","L 184 0 \n","L 1544 1831 \n","L 300 3500 \n","L 978 3500 \n","L 1906 2253 \n","L 2834 3500 \n","L 3513 3500 \n","z\n","\" transform=\"scale(0.015625)\"/>\n","      </defs>\n","      <use xlink:href=\"#DejaVuSans-78\"/>\n","     </g>\n","    </g>\n","   </g>\n","   <g id=\"matplotlib.axis_2\">\n","    <g id=\"ytick_1\">\n","     <g id=\"line2d_19\">\n","      <path d=\"M 43.78125 136.922786 \n","L 322.78125 136.922786 \n","\" clip-path=\"url(#p5bd91fec43)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n","     </g>\n","     <g id=\"line2d_20\">\n","      <defs>\n","       <path id=\"m87417e46c4\" d=\"M 0 0 \n","L -3.5 0 \n","\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n","      </defs>\n","      <g>\n","       <use xlink:href=\"#m87417e46c4\" x=\"43.78125\" y=\"136.922786\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n","      </g>\n","     </g>\n","     <g id=\"text_11\">\n","      <!-- 0.0 -->\n","      <g transform=\"translate(20.878125 140.722005)scale(0.1 -0.1)\">\n","       <defs>\n","        <path id=\"DejaVuSans-2e\" d=\"M 684 794 \n","L 1344 794 \n","L 1344 0 \n","L 684 0 \n","L 684 794 \n","z\n","\" transform=\"scale(0.015625)\"/>\n","       </defs>\n","       <use xlink:href=\"#DejaVuSans-30\"/>\n","       <use xlink:href=\"#DejaVuSans-2e\" x=\"63.623047\"/>\n","       <use xlink:href=\"#DejaVuSans-30\" x=\"95.410156\"/>\n","      </g>\n","     </g>\n","    </g>\n","    <g id=\"ytick_2\">\n","     <g id=\"line2d_21\">\n","      <path d=\"M 43.78125 112.213683 \n","L 322.78125 112.213683 \n","\" clip-path=\"url(#p5bd91fec43)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n","     </g>\n","     <g id=\"line2d_22\">\n","      <g>\n","       <use xlink:href=\"#m87417e46c4\" x=\"43.78125\" y=\"112.213683\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n","      </g>\n","     </g>\n","     <g id=\"text_12\">\n","      <!-- 0.2 -->\n","      <g transform=\"translate(20.878125 116.012902)scale(0.1 -0.1)\">\n","       <use xlink:href=\"#DejaVuSans-30\"/>\n","       <use xlink:href=\"#DejaVuSans-2e\" x=\"63.623047\"/>\n","       <use xlink:href=\"#DejaVuSans-32\" x=\"95.410156\"/>\n","      </g>\n","     </g>\n","    </g>\n","    <g id=\"ytick_3\">\n","     <g id=\"line2d_23\">\n","      <path d=\"M 43.78125 87.504581 \n","L 322.78125 87.504581 \n","\" clip-path=\"url(#p5bd91fec43)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n","     </g>\n","     <g id=\"line2d_24\">\n","      <g>\n","       <use xlink:href=\"#m87417e46c4\" x=\"43.78125\" y=\"87.504581\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n","      </g>\n","     </g>\n","     <g id=\"text_13\">\n","      <!-- 0.4 -->\n","      <g transform=\"translate(20.878125 91.3038)scale(0.1 -0.1)\">\n","       <use xlink:href=\"#DejaVuSans-30\"/>\n","       <use xlink:href=\"#DejaVuSans-2e\" x=\"63.623047\"/>\n","       <use xlink:href=\"#DejaVuSans-34\" x=\"95.410156\"/>\n","      </g>\n","     </g>\n","    </g>\n","    <g id=\"ytick_4\">\n","     <g id=\"line2d_25\">\n","      <path d=\"M 43.78125 62.795478 \n","L 322.78125 62.795478 \n","\" clip-path=\"url(#p5bd91fec43)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n","     </g>\n","     <g id=\"line2d_26\">\n","      <g>\n","       <use xlink:href=\"#m87417e46c4\" x=\"43.78125\" y=\"62.795478\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n","      </g>\n","     </g>\n","     <g id=\"text_14\">\n","      <!-- 0.6 -->\n","      <g transform=\"translate(20.878125 66.594697)scale(0.1 -0.1)\">\n","       <use xlink:href=\"#DejaVuSans-30\"/>\n","       <use xlink:href=\"#DejaVuSans-2e\" x=\"63.623047\"/>\n","       <use xlink:href=\"#DejaVuSans-36\" x=\"95.410156\"/>\n","      </g>\n","     </g>\n","    </g>\n","    <g id=\"ytick_5\">\n","     <g id=\"line2d_27\">\n","      <path d=\"M 43.78125 38.086375 \n","L 322.78125 38.086375 \n","\" clip-path=\"url(#p5bd91fec43)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n","     </g>\n","     <g id=\"line2d_28\">\n","      <g>\n","       <use xlink:href=\"#m87417e46c4\" x=\"43.78125\" y=\"38.086375\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n","      </g>\n","     </g>\n","     <g id=\"text_15\">\n","      <!-- 0.8 -->\n","      <g transform=\"translate(20.878125 41.885594)scale(0.1 -0.1)\">\n","       <use xlink:href=\"#DejaVuSans-30\"/>\n","       <use xlink:href=\"#DejaVuSans-2e\" x=\"63.623047\"/>\n","       <use xlink:href=\"#DejaVuSans-38\" x=\"95.410156\"/>\n","      </g>\n","     </g>\n","    </g>\n","    <g id=\"ytick_6\">\n","     <g id=\"line2d_29\">\n","      <path d=\"M 43.78125 13.377273 \n","L 322.78125 13.377273 \n","\" clip-path=\"url(#p5bd91fec43)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n","     </g>\n","     <g id=\"line2d_30\">\n","      <g>\n","       <use xlink:href=\"#m87417e46c4\" x=\"43.78125\" y=\"13.377273\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n","      </g>\n","     </g>\n","     <g id=\"text_16\">\n","      <!-- 1.0 -->\n","      <g transform=\"translate(20.878125 17.176491)scale(0.1 -0.1)\">\n","       <defs>\n","        <path id=\"DejaVuSans-31\" d=\"M 794 531 \n","L 1825 531 \n","L 1825 4091 \n","L 703 3866 \n","L 703 4441 \n","L 1819 4666 \n","L 2450 4666 \n","L 2450 531 \n","L 3481 531 \n","L 3481 0 \n","L 794 0 \n","L 794 531 \n","z\n","\" transform=\"scale(0.015625)\"/>\n","       </defs>\n","       <use xlink:href=\"#DejaVuSans-31\"/>\n","       <use xlink:href=\"#DejaVuSans-2e\" x=\"63.623047\"/>\n","       <use xlink:href=\"#DejaVuSans-30\" x=\"95.410156\"/>\n","      </g>\n","     </g>\n","    </g>\n","    <g id=\"text_17\">\n","     <!-- grad of tanh -->\n","     <g transform=\"translate(14.798438 105.977344)rotate(-90)scale(0.1 -0.1)\">\n","      <defs>\n","       <path id=\"DejaVuSans-67\" d=\"M 2906 1791 \n","Q 2906 2416 2648 2759 \n","Q 2391 3103 1925 3103 \n","Q 1463 3103 1205 2759 \n","Q 947 2416 947 1791 \n","Q 947 1169 1205 825 \n","Q 1463 481 1925 481 \n","Q 2391 481 2648 825 \n","Q 2906 1169 2906 1791 \n","z\n","M 3481 434 \n","Q 3481 -459 3084 -895 \n","Q 2688 -1331 1869 -1331 \n","Q 1566 -1331 1297 -1286 \n","Q 1028 -1241 775 -1147 \n","L 775 -588 \n","Q 1028 -725 1275 -790 \n","Q 1522 -856 1778 -856 \n","Q 2344 -856 2625 -561 \n","Q 2906 -266 2906 331 \n","L 2906 616 \n","Q 2728 306 2450 153 \n","Q 2172 0 1784 0 \n","Q 1141 0 747 490 \n","Q 353 981 353 1791 \n","Q 353 2603 747 3093 \n","Q 1141 3584 1784 3584 \n","Q 2172 3584 2450 3431 \n","Q 2728 3278 2906 2969 \n","L 2906 3500 \n","L 3481 3500 \n","L 3481 434 \n","z\n","\" transform=\"scale(0.015625)\"/>\n","       <path id=\"DejaVuSans-72\" d=\"M 2631 2963 \n","Q 2534 3019 2420 3045 \n","Q 2306 3072 2169 3072 \n","Q 1681 3072 1420 2755 \n","Q 1159 2438 1159 1844 \n","L 1159 0 \n","L 581 0 \n","L 581 3500 \n","L 1159 3500 \n","L 1159 2956 \n","Q 1341 3275 1631 3429 \n","Q 1922 3584 2338 3584 \n","Q 2397 3584 2469 3576 \n","Q 2541 3569 2628 3553 \n","L 2631 2963 \n","z\n","\" transform=\"scale(0.015625)\"/>\n","       <path id=\"DejaVuSans-61\" d=\"M 2194 1759 \n","Q 1497 1759 1228 1600 \n","Q 959 1441 959 1056 \n","Q 959 750 1161 570 \n","Q 1363 391 1709 391 \n","Q 2188 391 2477 730 \n","Q 2766 1069 2766 1631 \n","L 2766 1759 \n","L 2194 1759 \n","z\n","M 3341 1997 \n","L 3341 0 \n","L 2766 0 \n","L 2766 531 \n","Q 2569 213 2275 61 \n","Q 1981 -91 1556 -91 \n","Q 1019 -91 701 211 \n","Q 384 513 384 1019 \n","Q 384 1609 779 1909 \n","Q 1175 2209 1959 2209 \n","L 2766 2209 \n","L 2766 2266 \n","Q 2766 2663 2505 2880 \n","Q 2244 3097 1772 3097 \n","Q 1472 3097 1187 3025 \n","Q 903 2953 641 2809 \n","L 641 3341 \n","Q 956 3463 1253 3523 \n","Q 1550 3584 1831 3584 \n","Q 2591 3584 2966 3190 \n","Q 3341 2797 3341 1997 \n","z\n","\" transform=\"scale(0.015625)\"/>\n","       <path id=\"DejaVuSans-64\" d=\"M 2906 2969 \n","L 2906 4863 \n","L 3481 4863 \n","L 3481 0 \n","L 2906 0 \n","L 2906 525 \n","Q 2725 213 2448 61 \n","Q 2172 -91 1784 -91 \n","Q 1150 -91 751 415 \n","Q 353 922 353 1747 \n","Q 353 2572 751 3078 \n","Q 1150 3584 1784 3584 \n","Q 2172 3584 2448 3432 \n","Q 2725 3281 2906 2969 \n","z\n","M 947 1747 \n","Q 947 1113 1208 752 \n","Q 1469 391 1925 391 \n","Q 2381 391 2643 752 \n","Q 2906 1113 2906 1747 \n","Q 2906 2381 2643 2742 \n","Q 2381 3103 1925 3103 \n","Q 1469 3103 1208 2742 \n","Q 947 2381 947 1747 \n","z\n","\" transform=\"scale(0.015625)\"/>\n","       <path id=\"DejaVuSans-20\" transform=\"scale(0.015625)\"/>\n","       <path id=\"DejaVuSans-6f\" d=\"M 1959 3097 \n","Q 1497 3097 1228 2736 \n","Q 959 2375 959 1747 \n","Q 959 1119 1226 758 \n","Q 1494 397 1959 397 \n","Q 2419 397 2687 759 \n","Q 2956 1122 2956 1747 \n","Q 2956 2369 2687 2733 \n","Q 2419 3097 1959 3097 \n","z\n","M 1959 3584 \n","Q 2709 3584 3137 3096 \n","Q 3566 2609 3566 1747 \n","Q 3566 888 3137 398 \n","Q 2709 -91 1959 -91 \n","Q 1206 -91 779 398 \n","Q 353 888 353 1747 \n","Q 353 2609 779 3096 \n","Q 1206 3584 1959 3584 \n","z\n","\" transform=\"scale(0.015625)\"/>\n","       <path id=\"DejaVuSans-66\" d=\"M 2375 4863 \n","L 2375 4384 \n","L 1825 4384 \n","Q 1516 4384 1395 4259 \n","Q 1275 4134 1275 3809 \n","L 1275 3500 \n","L 2222 3500 \n","L 2222 3053 \n","L 1275 3053 \n","L 1275 0 \n","L 697 0 \n","L 697 3053 \n","L 147 3053 \n","L 147 3500 \n","L 697 3500 \n","L 697 3744 \n","Q 697 4328 969 4595 \n","Q 1241 4863 1831 4863 \n","L 2375 4863 \n","z\n","\" transform=\"scale(0.015625)\"/>\n","       <path id=\"DejaVuSans-74\" d=\"M 1172 4494 \n","L 1172 3500 \n","L 2356 3500 \n","L 2356 3053 \n","L 1172 3053 \n","L 1172 1153 \n","Q 1172 725 1289 603 \n","Q 1406 481 1766 481 \n","L 2356 481 \n","L 2356 0 \n","L 1766 0 \n","Q 1100 0 847 248 \n","Q 594 497 594 1153 \n","L 594 3053 \n","L 172 3053 \n","L 172 3500 \n","L 594 3500 \n","L 594 4494 \n","L 1172 4494 \n","z\n","\" transform=\"scale(0.015625)\"/>\n","       <path id=\"DejaVuSans-6e\" d=\"M 3513 2113 \n","L 3513 0 \n","L 2938 0 \n","L 2938 2094 \n","Q 2938 2591 2744 2837 \n","Q 2550 3084 2163 3084 \n","Q 1697 3084 1428 2787 \n","Q 1159 2491 1159 1978 \n","L 1159 0 \n","L 581 0 \n","L 581 3500 \n","L 1159 3500 \n","L 1159 2956 \n","Q 1366 3272 1645 3428 \n","Q 1925 3584 2291 3584 \n","Q 2894 3584 3203 3211 \n","Q 3513 2838 3513 2113 \n","z\n","\" transform=\"scale(0.015625)\"/>\n","       <path id=\"DejaVuSans-68\" d=\"M 3513 2113 \n","L 3513 0 \n","L 2938 0 \n","L 2938 2094 \n","Q 2938 2591 2744 2837 \n","Q 2550 3084 2163 3084 \n","Q 1697 3084 1428 2787 \n","Q 1159 2491 1159 1978 \n","L 1159 0 \n","L 581 0 \n","L 581 4863 \n","L 1159 4863 \n","L 1159 2956 \n","Q 1366 3272 1645 3428 \n","Q 1925 3584 2291 3584 \n","Q 2894 3584 3203 3211 \n","Q 3513 2838 3513 2113 \n","z\n","\" transform=\"scale(0.015625)\"/>\n","      </defs>\n","      <use xlink:href=\"#DejaVuSans-67\"/>\n","      <use xlink:href=\"#DejaVuSans-72\" x=\"63.476562\"/>\n","      <use xlink:href=\"#DejaVuSans-61\" x=\"104.589844\"/>\n","      <use xlink:href=\"#DejaVuSans-64\" x=\"165.869141\"/>\n","      <use xlink:href=\"#DejaVuSans-20\" x=\"229.345703\"/>\n","      <use xlink:href=\"#DejaVuSans-6f\" x=\"261.132812\"/>\n","      <use xlink:href=\"#DejaVuSans-66\" x=\"322.314453\"/>\n","      <use xlink:href=\"#DejaVuSans-20\" x=\"357.519531\"/>\n","      <use xlink:href=\"#DejaVuSans-74\" x=\"389.306641\"/>\n","      <use xlink:href=\"#DejaVuSans-61\" x=\"428.515625\"/>\n","      <use xlink:href=\"#DejaVuSans-6e\" x=\"489.794922\"/>\n","      <use xlink:href=\"#DejaVuSans-68\" x=\"553.173828\"/>\n","     </g>\n","    </g>\n","   </g>\n","   <g id=\"line2d_31\">\n","    <path d=\"M 56.463068 136.922727 \n","L 117.080561 136.811705 \n","L 126.651751 136.554381 \n","L 131.437342 136.252347 \n","L 136.222933 135.703888 \n","L 139.413328 135.108803 \n","L 142.60372 134.226425 \n","L 144.198919 133.637433 \n","L 145.794115 132.921923 \n","L 147.389311 132.053728 \n","L 148.98451 131.001821 \n","L 150.579706 129.729595 \n","L 152.174906 128.194195 \n","L 153.770102 126.346073 \n","L 155.365299 124.128614 \n","L 156.960497 121.478376 \n","L 158.555695 118.325836 \n","L 160.150892 114.597294 \n","L 161.74609 110.218121 \n","L 163.341286 105.118076 \n","L 164.936483 99.23894 \n","L 166.531681 92.545076 \n","L 168.126879 85.036846 \n","L 169.722075 76.766329 \n","L 172.912471 58.503559 \n","L 176.102865 39.760701 \n","L 177.698062 31.212451 \n","L 179.29326 23.861724 \n","L 180.888457 18.190235 \n","L 182.483654 14.604538 \n","L 184.078852 13.377273 \n","L 185.674049 14.604538 \n","L 187.269246 18.190235 \n","L 188.864444 23.861724 \n","L 190.459641 31.212451 \n","L 192.054838 39.760701 \n","L 198.435628 76.766329 \n","L 200.030825 85.036846 \n","L 201.626022 92.545076 \n","L 203.22122 99.23894 \n","L 204.816416 105.118061 \n","L 206.411613 110.218121 \n","L 208.006811 114.597294 \n","L 209.602009 118.325836 \n","L 211.197206 121.478376 \n","L 212.792404 124.128614 \n","L 214.3876 126.346059 \n","L 215.982798 128.194195 \n","L 217.577993 129.729595 \n","L 219.173193 131.001821 \n","L 220.768389 132.053728 \n","L 222.363588 132.921923 \n","L 223.958784 133.637433 \n","L 227.149179 134.710798 \n","L 230.339575 135.435645 \n","L 233.52997 135.923943 \n","L 238.315561 136.373595 \n","L 244.696348 136.675719 \n","L 254.267534 136.848319 \n","L 275.005094 136.917249 \n","L 310.099432 136.922713 \n","L 310.099432 136.922713 \n","\" clip-path=\"url(#p5bd91fec43)\" style=\"fill: none; stroke: #1f77b4; stroke-width: 1.5; stroke-linecap: square\"/>\n","   </g>\n","   <g id=\"patch_3\">\n","    <path d=\"M 43.78125 143.1 \n","L 43.78125 7.2 \n","\" style=\"fill: none; stroke: #000000; stroke-width: 0.8; stroke-linejoin: miter; stroke-linecap: square\"/>\n","   </g>\n","   <g id=\"patch_4\">\n","    <path d=\"M 322.78125 143.1 \n","L 322.78125 7.2 \n","\" style=\"fill: none; stroke: #000000; stroke-width: 0.8; stroke-linejoin: miter; stroke-linecap: square\"/>\n","   </g>\n","   <g id=\"patch_5\">\n","    <path d=\"M 43.78125 143.1 \n","L 322.78125 143.1 \n","\" style=\"fill: none; stroke: #000000; stroke-width: 0.8; stroke-linejoin: miter; stroke-linecap: square\"/>\n","   </g>\n","   <g id=\"patch_6\">\n","    <path d=\"M 43.78125 7.2 \n","L 322.78125 7.2 \n","\" style=\"fill: none; stroke: #000000; stroke-width: 0.8; stroke-linejoin: miter; stroke-linecap: square\"/>\n","   </g>\n","  </g>\n"," </g>\n"," <defs>\n","  <clipPath id=\"p5bd91fec43\">\n","   <rect x=\"43.78125\" y=\"7.2\" width=\"279\" height=\"135.9\"/>\n","  </clipPath>\n"," </defs>\n","</svg>\n"],"text/plain":["<Figure size 360x180 with 1 Axes>"]},"metadata":{"needs_background":"light"},"output_type":"display_data"}],"source":["# 清除以前的梯度\n","x.grad.data.zero_()\n","y.backward(torch.ones_like(x),retain_graph=True)\n","d2l.plot(x.detach(), x.grad, 'x', 'grad of tanh', figsize=(5, 2.5))"]},{"cell_type":"markdown","metadata":{"origin_pos":0,"id":"4d522efa"},"source":["# 多层感知机的简洁实现\n",":label:`sec_mlp_concise`\n","\n","正如你所期待的，我们可以(**通过高级API更简洁地实现多层感知机**)。\n"],"id":"4d522efa"},{"cell_type":"code","execution_count":null,"metadata":{"execution":{"iopub.execute_input":"2022-07-31T02:29:21.553674Z","iopub.status.busy":"2022-07-31T02:29:21.553096Z","iopub.status.idle":"2022-07-31T02:29:23.419894Z","shell.execute_reply":"2022-07-31T02:29:23.419131Z"},"origin_pos":2,"tab":["pytorch"],"id":"be821b15"},"outputs":[],"source":["import torch\n","from torch import nn\n","from d2l import torch as d2l"],"id":"be821b15"},{"cell_type":"markdown","metadata":{"origin_pos":4,"id":"53e81ff4"},"source":["## 模型\n","\n","与softmax回归的简洁实现（ :numref:`sec_softmax_concise`）相比，\n","唯一的区别是我们添加了2个全连接层（之前我们只添加了1个全连接层）。\n","第一层是[**隐藏层**]，它(**包含256个隐藏单元，并使用了ReLU激活函数**)。\n","第二层是输出层。\n"],"id":"53e81ff4"},{"cell_type":"code","execution_count":null,"metadata":{"execution":{"iopub.execute_input":"2022-07-31T02:29:23.424678Z","iopub.status.busy":"2022-07-31T02:29:23.424361Z","iopub.status.idle":"2022-07-31T02:29:23.433077Z","shell.execute_reply":"2022-07-31T02:29:23.432448Z"},"origin_pos":6,"tab":["pytorch"],"id":"5e3c3d86"},"outputs":[],"source":["net = nn.Sequential(nn.Flatten(),\n","                    nn.Linear(784, 256),\n","                    nn.ReLU(),\n","                    nn.Linear(256, 10))\n","\n","def init_weights(m):\n","    if type(m) == nn.Linear:\n","        nn.init.normal_(m.weight, std=0.01)\n","\n","net.apply(init_weights);"],"id":"5e3c3d86"},{"cell_type":"markdown","metadata":{"origin_pos":8,"id":"11d405e0"},"source":["[**训练过程**]的实现与我们实现softmax回归时完全相同，\n","这种模块化设计使我们能够将与模型架构有关的内容独立出来。\n"],"id":"11d405e0"},{"cell_type":"code","execution_count":null,"metadata":{"execution":{"iopub.execute_input":"2022-07-31T02:29:23.437358Z","iopub.status.busy":"2022-07-31T02:29:23.436999Z","iopub.status.idle":"2022-07-31T02:29:23.440994Z","shell.execute_reply":"2022-07-31T02:29:23.440346Z"},"origin_pos":10,"tab":["pytorch"],"id":"abfd92d5"},"outputs":[],"source":["batch_size, lr, num_epochs = 256, 0.1, 10\n","loss = nn.CrossEntropyLoss(reduction='none')\n","trainer = torch.optim.SGD(net.parameters(), lr=lr)"],"id":"abfd92d5"},{"cell_type":"code","execution_count":null,"metadata":{"execution":{"iopub.execute_input":"2022-07-31T02:29:23.445300Z","iopub.status.busy":"2022-07-31T02:29:23.444959Z","iopub.status.idle":"2022-07-31T02:29:57.656280Z","shell.execute_reply":"2022-07-31T02:29:57.655475Z"},"origin_pos":12,"tab":["pytorch"],"id":"cfef131f","outputId":"0855b85e-a28a-4894-a337-80c1ab80ea64"},"outputs":[{"data":{"image/svg+xml":["<?xml version=\"1.0\" encoding=\"utf-8\" standalone=\"no\"?>\n","<!DOCTYPE svg PUBLIC \"-//W3C//DTD SVG 1.1//EN\"\n","  \"http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd\">\n","<svg xmlns:xlink=\"http://www.w3.org/1999/xlink\" width=\"238.965625pt\" height=\"180.65625pt\" viewBox=\"0 0 238.965625 180.65625\" xmlns=\"http://www.w3.org/2000/svg\" version=\"1.1\">\n"," <metadata>\n","  <rdf:RDF xmlns:dc=\"http://purl.org/dc/elements/1.1/\" xmlns:cc=\"http://creativecommons.org/ns#\" xmlns:rdf=\"http://www.w3.org/1999/02/22-rdf-syntax-ns#\">\n","   <cc:Work>\n","    <dc:type rdf:resource=\"http://purl.org/dc/dcmitype/StillImage\"/>\n","    <dc:date>2022-07-31T02:29:57.616205</dc:date>\n","    <dc:format>image/svg+xml</dc:format>\n","    <dc:creator>\n","     <cc:Agent>\n","      <dc:title>Matplotlib v3.5.1, https://matplotlib.org/</dc:title>\n","     </cc:Agent>\n","    </dc:creator>\n","   </cc:Work>\n","  </rdf:RDF>\n"," </metadata>\n"," <defs>\n","  <style type=\"text/css\">*{stroke-linejoin: round; stroke-linecap: butt}</style>\n"," </defs>\n"," <g id=\"figure_1\">\n","  <g id=\"patch_1\">\n","   <path d=\"M 0 180.65625 \n","L 238.965625 180.65625 \n","L 238.965625 0 \n","L 0 0 \n","L 0 180.65625 \n","z\n","\" style=\"fill: none\"/>\n","  </g>\n","  <g id=\"axes_1\">\n","   <g id=\"patch_2\">\n","    <path d=\"M 30.103125 143.1 \n","L 225.403125 143.1 \n","L 225.403125 7.2 \n","L 30.103125 7.2 \n","z\n","\" style=\"fill: #ffffff\"/>\n","   </g>\n","   <g id=\"matplotlib.axis_1\">\n","    <g id=\"xtick_1\">\n","     <g id=\"line2d_1\">\n","      <path d=\"M 51.803125 143.1 \n","L 51.803125 7.2 \n","\" clip-path=\"url(#p1f1a59b8d9)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n","     </g>\n","     <g id=\"line2d_2\">\n","      <defs>\n","       <path id=\"m87f692bb68\" d=\"M 0 0 \n","L 0 3.5 \n","\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n","      </defs>\n","      <g>\n","       <use xlink:href=\"#m87f692bb68\" x=\"51.803125\" y=\"143.1\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n","      </g>\n","     </g>\n","     <g id=\"text_1\">\n","      <!-- 2 -->\n","      <g transform=\"translate(48.621875 157.698438)scale(0.1 -0.1)\">\n","       <defs>\n","        <path id=\"DejaVuSans-32\" d=\"M 1228 531 \n","L 3431 531 \n","L 3431 0 \n","L 469 0 \n","L 469 531 \n","Q 828 903 1448 1529 \n","Q 2069 2156 2228 2338 \n","Q 2531 2678 2651 2914 \n","Q 2772 3150 2772 3378 \n","Q 2772 3750 2511 3984 \n","Q 2250 4219 1831 4219 \n","Q 1534 4219 1204 4116 \n","Q 875 4013 500 3803 \n","L 500 4441 \n","Q 881 4594 1212 4672 \n","Q 1544 4750 1819 4750 \n","Q 2544 4750 2975 4387 \n","Q 3406 4025 3406 3419 \n","Q 3406 3131 3298 2873 \n","Q 3191 2616 2906 2266 \n","Q 2828 2175 2409 1742 \n","Q 1991 1309 1228 531 \n","z\n","\" transform=\"scale(0.015625)\"/>\n","       </defs>\n","       <use xlink:href=\"#DejaVuSans-32\"/>\n","      </g>\n","     </g>\n","    </g>\n","    <g id=\"xtick_2\">\n","     <g id=\"line2d_3\">\n","      <path d=\"M 95.203125 143.1 \n","L 95.203125 7.2 \n","\" clip-path=\"url(#p1f1a59b8d9)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n","     </g>\n","     <g id=\"line2d_4\">\n","      <g>\n","       <use xlink:href=\"#m87f692bb68\" x=\"95.203125\" y=\"143.1\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n","      </g>\n","     </g>\n","     <g id=\"text_2\">\n","      <!-- 4 -->\n","      <g transform=\"translate(92.021875 157.698438)scale(0.1 -0.1)\">\n","       <defs>\n","        <path id=\"DejaVuSans-34\" d=\"M 2419 4116 \n","L 825 1625 \n","L 2419 1625 \n","L 2419 4116 \n","z\n","M 2253 4666 \n","L 3047 4666 \n","L 3047 1625 \n","L 3713 1625 \n","L 3713 1100 \n","L 3047 1100 \n","L 3047 0 \n","L 2419 0 \n","L 2419 1100 \n","L 313 1100 \n","L 313 1709 \n","L 2253 4666 \n","z\n","\" transform=\"scale(0.015625)\"/>\n","       </defs>\n","       <use xlink:href=\"#DejaVuSans-34\"/>\n","      </g>\n","     </g>\n","    </g>\n","    <g id=\"xtick_3\">\n","     <g id=\"line2d_5\">\n","      <path d=\"M 138.603125 143.1 \n","L 138.603125 7.2 \n","\" clip-path=\"url(#p1f1a59b8d9)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n","     </g>\n","     <g id=\"line2d_6\">\n","      <g>\n","       <use xlink:href=\"#m87f692bb68\" x=\"138.603125\" y=\"143.1\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n","      </g>\n","     </g>\n","     <g id=\"text_3\">\n","      <!-- 6 -->\n","      <g transform=\"translate(135.421875 157.698438)scale(0.1 -0.1)\">\n","       <defs>\n","        <path id=\"DejaVuSans-36\" d=\"M 2113 2584 \n","Q 1688 2584 1439 2293 \n","Q 1191 2003 1191 1497 \n","Q 1191 994 1439 701 \n","Q 1688 409 2113 409 \n","Q 2538 409 2786 701 \n","Q 3034 994 3034 1497 \n","Q 3034 2003 2786 2293 \n","Q 2538 2584 2113 2584 \n","z\n","M 3366 4563 \n","L 3366 3988 \n","Q 3128 4100 2886 4159 \n","Q 2644 4219 2406 4219 \n","Q 1781 4219 1451 3797 \n","Q 1122 3375 1075 2522 \n","Q 1259 2794 1537 2939 \n","Q 1816 3084 2150 3084 \n","Q 2853 3084 3261 2657 \n","Q 3669 2231 3669 1497 \n","Q 3669 778 3244 343 \n","Q 2819 -91 2113 -91 \n","Q 1303 -91 875 529 \n","Q 447 1150 447 2328 \n","Q 447 3434 972 4092 \n","Q 1497 4750 2381 4750 \n","Q 2619 4750 2861 4703 \n","Q 3103 4656 3366 4563 \n","z\n","\" transform=\"scale(0.015625)\"/>\n","       </defs>\n","       <use xlink:href=\"#DejaVuSans-36\"/>\n","      </g>\n","     </g>\n","    </g>\n","    <g id=\"xtick_4\">\n","     <g id=\"line2d_7\">\n","      <path d=\"M 182.003125 143.1 \n","L 182.003125 7.2 \n","\" clip-path=\"url(#p1f1a59b8d9)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n","     </g>\n","     <g id=\"line2d_8\">\n","      <g>\n","       <use xlink:href=\"#m87f692bb68\" x=\"182.003125\" y=\"143.1\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n","      </g>\n","     </g>\n","     <g id=\"text_4\">\n","      <!-- 8 -->\n","      <g transform=\"translate(178.821875 157.698438)scale(0.1 -0.1)\">\n","       <defs>\n","        <path id=\"DejaVuSans-38\" d=\"M 2034 2216 \n","Q 1584 2216 1326 1975 \n","Q 1069 1734 1069 1313 \n","Q 1069 891 1326 650 \n","Q 1584 409 2034 409 \n","Q 2484 409 2743 651 \n","Q 3003 894 3003 1313 \n","Q 3003 1734 2745 1975 \n","Q 2488 2216 2034 2216 \n","z\n","M 1403 2484 \n","Q 997 2584 770 2862 \n","Q 544 3141 544 3541 \n","Q 544 4100 942 4425 \n","Q 1341 4750 2034 4750 \n","Q 2731 4750 3128 4425 \n","Q 3525 4100 3525 3541 \n","Q 3525 3141 3298 2862 \n","Q 3072 2584 2669 2484 \n","Q 3125 2378 3379 2068 \n","Q 3634 1759 3634 1313 \n","Q 3634 634 3220 271 \n","Q 2806 -91 2034 -91 \n","Q 1263 -91 848 271 \n","Q 434 634 434 1313 \n","Q 434 1759 690 2068 \n","Q 947 2378 1403 2484 \n","z\n","M 1172 3481 \n","Q 1172 3119 1398 2916 \n","Q 1625 2713 2034 2713 \n","Q 2441 2713 2670 2916 \n","Q 2900 3119 2900 3481 \n","Q 2900 3844 2670 4047 \n","Q 2441 4250 2034 4250 \n","Q 1625 4250 1398 4047 \n","Q 1172 3844 1172 3481 \n","z\n","\" transform=\"scale(0.015625)\"/>\n","       </defs>\n","       <use xlink:href=\"#DejaVuSans-38\"/>\n","      </g>\n","     </g>\n","    </g>\n","    <g id=\"xtick_5\">\n","     <g id=\"line2d_9\">\n","      <path d=\"M 225.403125 143.1 \n","L 225.403125 7.2 \n","\" clip-path=\"url(#p1f1a59b8d9)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n","     </g>\n","     <g id=\"line2d_10\">\n","      <g>\n","       <use xlink:href=\"#m87f692bb68\" x=\"225.403125\" y=\"143.1\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n","      </g>\n","     </g>\n","     <g id=\"text_5\">\n","      <!-- 10 -->\n","      <g transform=\"translate(219.040625 157.698438)scale(0.1 -0.1)\">\n","       <defs>\n","        <path id=\"DejaVuSans-31\" d=\"M 794 531 \n","L 1825 531 \n","L 1825 4091 \n","L 703 3866 \n","L 703 4441 \n","L 1819 4666 \n","L 2450 4666 \n","L 2450 531 \n","L 3481 531 \n","L 3481 0 \n","L 794 0 \n","L 794 531 \n","z\n","\" transform=\"scale(0.015625)\"/>\n","        <path id=\"DejaVuSans-30\" d=\"M 2034 4250 \n","Q 1547 4250 1301 3770 \n","Q 1056 3291 1056 2328 \n","Q 1056 1369 1301 889 \n","Q 1547 409 2034 409 \n","Q 2525 409 2770 889 \n","Q 3016 1369 3016 2328 \n","Q 3016 3291 2770 3770 \n","Q 2525 4250 2034 4250 \n","z\n","M 2034 4750 \n","Q 2819 4750 3233 4129 \n","Q 3647 3509 3647 2328 \n","Q 3647 1150 3233 529 \n","Q 2819 -91 2034 -91 \n","Q 1250 -91 836 529 \n","Q 422 1150 422 2328 \n","Q 422 3509 836 4129 \n","Q 1250 4750 2034 4750 \n","z\n","\" transform=\"scale(0.015625)\"/>\n","       </defs>\n","       <use xlink:href=\"#DejaVuSans-31\"/>\n","       <use xlink:href=\"#DejaVuSans-30\" x=\"63.623047\"/>\n","      </g>\n","     </g>\n","    </g>\n","    <g id=\"text_6\">\n","     <!-- epoch -->\n","     <g transform=\"translate(112.525 171.376563)scale(0.1 -0.1)\">\n","      <defs>\n","       <path id=\"DejaVuSans-65\" d=\"M 3597 1894 \n","L 3597 1613 \n","L 953 1613 \n","Q 991 1019 1311 708 \n","Q 1631 397 2203 397 \n","Q 2534 397 2845 478 \n","Q 3156 559 3463 722 \n","L 3463 178 \n","Q 3153 47 2828 -22 \n","Q 2503 -91 2169 -91 \n","Q 1331 -91 842 396 \n","Q 353 884 353 1716 \n","Q 353 2575 817 3079 \n","Q 1281 3584 2069 3584 \n","Q 2775 3584 3186 3129 \n","Q 3597 2675 3597 1894 \n","z\n","M 3022 2063 \n","Q 3016 2534 2758 2815 \n","Q 2500 3097 2075 3097 \n","Q 1594 3097 1305 2825 \n","Q 1016 2553 972 2059 \n","L 3022 2063 \n","z\n","\" transform=\"scale(0.015625)\"/>\n","       <path id=\"DejaVuSans-70\" d=\"M 1159 525 \n","L 1159 -1331 \n","L 581 -1331 \n","L 581 3500 \n","L 1159 3500 \n","L 1159 2969 \n","Q 1341 3281 1617 3432 \n","Q 1894 3584 2278 3584 \n","Q 2916 3584 3314 3078 \n","Q 3713 2572 3713 1747 \n","Q 3713 922 3314 415 \n","Q 2916 -91 2278 -91 \n","Q 1894 -91 1617 61 \n","Q 1341 213 1159 525 \n","z\n","M 3116 1747 \n","Q 3116 2381 2855 2742 \n","Q 2594 3103 2138 3103 \n","Q 1681 3103 1420 2742 \n","Q 1159 2381 1159 1747 \n","Q 1159 1113 1420 752 \n","Q 1681 391 2138 391 \n","Q 2594 391 2855 752 \n","Q 3116 1113 3116 1747 \n","z\n","\" transform=\"scale(0.015625)\"/>\n","       <path id=\"DejaVuSans-6f\" d=\"M 1959 3097 \n","Q 1497 3097 1228 2736 \n","Q 959 2375 959 1747 \n","Q 959 1119 1226 758 \n","Q 1494 397 1959 397 \n","Q 2419 397 2687 759 \n","Q 2956 1122 2956 1747 \n","Q 2956 2369 2687 2733 \n","Q 2419 3097 1959 3097 \n","z\n","M 1959 3584 \n","Q 2709 3584 3137 3096 \n","Q 3566 2609 3566 1747 \n","Q 3566 888 3137 398 \n","Q 2709 -91 1959 -91 \n","Q 1206 -91 779 398 \n","Q 353 888 353 1747 \n","Q 353 2609 779 3096 \n","Q 1206 3584 1959 3584 \n","z\n","\" transform=\"scale(0.015625)\"/>\n","       <path id=\"DejaVuSans-63\" d=\"M 3122 3366 \n","L 3122 2828 \n","Q 2878 2963 2633 3030 \n","Q 2388 3097 2138 3097 \n","Q 1578 3097 1268 2742 \n","Q 959 2388 959 1747 \n","Q 959 1106 1268 751 \n","Q 1578 397 2138 397 \n","Q 2388 397 2633 464 \n","Q 2878 531 3122 666 \n","L 3122 134 \n","Q 2881 22 2623 -34 \n","Q 2366 -91 2075 -91 \n","Q 1284 -91 818 406 \n","Q 353 903 353 1747 \n","Q 353 2603 823 3093 \n","Q 1294 3584 2113 3584 \n","Q 2378 3584 2631 3529 \n","Q 2884 3475 3122 3366 \n","z\n","\" transform=\"scale(0.015625)\"/>\n","       <path id=\"DejaVuSans-68\" d=\"M 3513 2113 \n","L 3513 0 \n","L 2938 0 \n","L 2938 2094 \n","Q 2938 2591 2744 2837 \n","Q 2550 3084 2163 3084 \n","Q 1697 3084 1428 2787 \n","Q 1159 2491 1159 1978 \n","L 1159 0 \n","L 581 0 \n","L 581 4863 \n","L 1159 4863 \n","L 1159 2956 \n","Q 1366 3272 1645 3428 \n","Q 1925 3584 2291 3584 \n","Q 2894 3584 3203 3211 \n","Q 3513 2838 3513 2113 \n","z\n","\" transform=\"scale(0.015625)\"/>\n","      </defs>\n","      <use xlink:href=\"#DejaVuSans-65\"/>\n","      <use xlink:href=\"#DejaVuSans-70\" x=\"61.523438\"/>\n","      <use xlink:href=\"#DejaVuSans-6f\" x=\"125\"/>\n","      <use xlink:href=\"#DejaVuSans-63\" x=\"186.181641\"/>\n","      <use xlink:href=\"#DejaVuSans-68\" x=\"241.162109\"/>\n","     </g>\n","    </g>\n","   </g>\n","   <g id=\"matplotlib.axis_2\">\n","    <g id=\"ytick_1\">\n","     <g id=\"line2d_11\">\n","      <path d=\"M 30.103125 120.45 \n","L 225.403125 120.45 \n","\" clip-path=\"url(#p1f1a59b8d9)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n","     </g>\n","     <g id=\"line2d_12\">\n","      <defs>\n","       <path id=\"mfbf14b93b2\" d=\"M 0 0 \n","L -3.5 0 \n","\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n","      </defs>\n","      <g>\n","       <use xlink:href=\"#mfbf14b93b2\" x=\"30.103125\" y=\"120.45\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n","      </g>\n","     </g>\n","     <g id=\"text_7\">\n","      <!-- 0.4 -->\n","      <g transform=\"translate(7.2 124.249219)scale(0.1 -0.1)\">\n","       <defs>\n","        <path id=\"DejaVuSans-2e\" d=\"M 684 794 \n","L 1344 794 \n","L 1344 0 \n","L 684 0 \n","L 684 794 \n","z\n","\" transform=\"scale(0.015625)\"/>\n","       </defs>\n","       <use xlink:href=\"#DejaVuSans-30\"/>\n","       <use xlink:href=\"#DejaVuSans-2e\" x=\"63.623047\"/>\n","       <use xlink:href=\"#DejaVuSans-34\" x=\"95.410156\"/>\n","      </g>\n","     </g>\n","    </g>\n","    <g id=\"ytick_2\">\n","     <g id=\"line2d_13\">\n","      <path d=\"M 30.103125 75.15 \n","L 225.403125 75.15 \n","\" clip-path=\"url(#p1f1a59b8d9)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n","     </g>\n","     <g id=\"line2d_14\">\n","      <g>\n","       <use xlink:href=\"#mfbf14b93b2\" x=\"30.103125\" y=\"75.15\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n","      </g>\n","     </g>\n","     <g id=\"text_8\">\n","      <!-- 0.6 -->\n","      <g transform=\"translate(7.2 78.949219)scale(0.1 -0.1)\">\n","       <use xlink:href=\"#DejaVuSans-30\"/>\n","       <use xlink:href=\"#DejaVuSans-2e\" x=\"63.623047\"/>\n","       <use xlink:href=\"#DejaVuSans-36\" x=\"95.410156\"/>\n","      </g>\n","     </g>\n","    </g>\n","    <g id=\"ytick_3\">\n","     <g id=\"line2d_15\">\n","      <path d=\"M 30.103125 29.85 \n","L 225.403125 29.85 \n","\" clip-path=\"url(#p1f1a59b8d9)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n","     </g>\n","     <g id=\"line2d_16\">\n","      <g>\n","       <use xlink:href=\"#mfbf14b93b2\" x=\"30.103125\" y=\"29.85\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n","      </g>\n","     </g>\n","     <g id=\"text_9\">\n","      <!-- 0.8 -->\n","      <g transform=\"translate(7.2 33.649219)scale(0.1 -0.1)\">\n","       <use xlink:href=\"#DejaVuSans-30\"/>\n","       <use xlink:href=\"#DejaVuSans-2e\" x=\"63.623047\"/>\n","       <use xlink:href=\"#DejaVuSans-38\" x=\"95.410156\"/>\n","      </g>\n","     </g>\n","    </g>\n","   </g>\n","   <g id=\"line2d_17\">\n","    <path d=\"M 35.085071 -1 \n","L 51.803125 76.810298 \n","L 73.503125 93.142551 \n","L 95.203125 102.471887 \n","L 116.903125 108.512262 \n","L 138.603125 112.745328 \n","L 160.303125 116.733151 \n","L 182.003125 118.982144 \n","L 203.703125 121.986568 \n","L 225.403125 124.066666 \n","\" clip-path=\"url(#p1f1a59b8d9)\" style=\"fill: none; stroke: #1f77b4; stroke-width: 1.5; stroke-linecap: square\"/>\n","   </g>\n","   <g id=\"line2d_18\">\n","    <path d=\"M 30.103125 66.697775 \n","L 51.803125 31.439275 \n","L 73.503125 25.88625 \n","L 95.203125 22.507625 \n","L 116.903125 20.476675 \n","L 138.603125 19.25735 \n","L 160.303125 17.720925 \n","L 182.003125 16.84135 \n","L 203.703125 15.991975 \n","L 225.403125 15.297375 \n","\" clip-path=\"url(#p1f1a59b8d9)\" style=\"fill: none; stroke-dasharray: 5.55,2.4; stroke-dashoffset: 0; stroke: #bf00bf; stroke-width: 1.5\"/>\n","   </g>\n","   <g id=\"line2d_19\">\n","    <path d=\"M 30.103125 51.6393 \n","L 51.803125 29.82735 \n","L 73.503125 27.97005 \n","L 95.203125 32.8398 \n","L 116.903125 30.303 \n","L 138.603125 21.3789 \n","L 160.303125 22.1037 \n","L 182.003125 19.9293 \n","L 203.703125 18.7062 \n","L 225.403125 19.90665 \n","\" clip-path=\"url(#p1f1a59b8d9)\" style=\"fill: none; stroke-dasharray: 9.6,2.4,1.5,2.4; stroke-dashoffset: 0; stroke: #008000; stroke-width: 1.5\"/>\n","   </g>\n","   <g id=\"patch_3\">\n","    <path d=\"M 30.103125 143.1 \n","L 30.103125 7.2 \n","\" style=\"fill: none; stroke: #000000; stroke-width: 0.8; stroke-linejoin: miter; stroke-linecap: square\"/>\n","   </g>\n","   <g id=\"patch_4\">\n","    <path d=\"M 225.403125 143.1 \n","L 225.403125 7.2 \n","\" style=\"fill: none; stroke: #000000; stroke-width: 0.8; stroke-linejoin: miter; stroke-linecap: square\"/>\n","   </g>\n","   <g id=\"patch_5\">\n","    <path d=\"M 30.103125 143.1 \n","L 225.403125 143.1 \n","\" style=\"fill: none; stroke: #000000; stroke-width: 0.8; stroke-linejoin: miter; stroke-linecap: square\"/>\n","   </g>\n","   <g id=\"patch_6\">\n","    <path d=\"M 30.103125 7.2 \n","L 225.403125 7.2 \n","\" style=\"fill: none; stroke: #000000; stroke-width: 0.8; stroke-linejoin: miter; stroke-linecap: square\"/>\n","   </g>\n","   <g id=\"legend_1\">\n","    <g id=\"patch_7\">\n","     <path d=\"M 140.634375 98.667187 \n","L 218.403125 98.667187 \n","Q 220.403125 98.667187 220.403125 96.667187 \n","L 220.403125 53.632812 \n","Q 220.403125 51.632812 218.403125 51.632812 \n","L 140.634375 51.632812 \n","Q 138.634375 51.632812 138.634375 53.632812 \n","L 138.634375 96.667187 \n","Q 138.634375 98.667187 140.634375 98.667187 \n","z\n","\" style=\"fill: #ffffff; opacity: 0.8; stroke: #cccccc; stroke-linejoin: miter\"/>\n","    </g>\n","    <g id=\"line2d_20\">\n","     <path d=\"M 142.634375 59.73125 \n","L 152.634375 59.73125 \n","L 162.634375 59.73125 \n","\" style=\"fill: none; stroke: #1f77b4; stroke-width: 1.5; stroke-linecap: square\"/>\n","    </g>\n","    <g id=\"text_10\">\n","     <!-- train loss -->\n","     <g transform=\"translate(170.634375 63.23125)scale(0.1 -0.1)\">\n","      <defs>\n","       <path id=\"DejaVuSans-74\" d=\"M 1172 4494 \n","L 1172 3500 \n","L 2356 3500 \n","L 2356 3053 \n","L 1172 3053 \n","L 1172 1153 \n","Q 1172 725 1289 603 \n","Q 1406 481 1766 481 \n","L 2356 481 \n","L 2356 0 \n","L 1766 0 \n","Q 1100 0 847 248 \n","Q 594 497 594 1153 \n","L 594 3053 \n","L 172 3053 \n","L 172 3500 \n","L 594 3500 \n","L 594 4494 \n","L 1172 4494 \n","z\n","\" transform=\"scale(0.015625)\"/>\n","       <path id=\"DejaVuSans-72\" d=\"M 2631 2963 \n","Q 2534 3019 2420 3045 \n","Q 2306 3072 2169 3072 \n","Q 1681 3072 1420 2755 \n","Q 1159 2438 1159 1844 \n","L 1159 0 \n","L 581 0 \n","L 581 3500 \n","L 1159 3500 \n","L 1159 2956 \n","Q 1341 3275 1631 3429 \n","Q 1922 3584 2338 3584 \n","Q 2397 3584 2469 3576 \n","Q 2541 3569 2628 3553 \n","L 2631 2963 \n","z\n","\" transform=\"scale(0.015625)\"/>\n","       <path id=\"DejaVuSans-61\" d=\"M 2194 1759 \n","Q 1497 1759 1228 1600 \n","Q 959 1441 959 1056 \n","Q 959 750 1161 570 \n","Q 1363 391 1709 391 \n","Q 2188 391 2477 730 \n","Q 2766 1069 2766 1631 \n","L 2766 1759 \n","L 2194 1759 \n","z\n","M 3341 1997 \n","L 3341 0 \n","L 2766 0 \n","L 2766 531 \n","Q 2569 213 2275 61 \n","Q 1981 -91 1556 -91 \n","Q 1019 -91 701 211 \n","Q 384 513 384 1019 \n","Q 384 1609 779 1909 \n","Q 1175 2209 1959 2209 \n","L 2766 2209 \n","L 2766 2266 \n","Q 2766 2663 2505 2880 \n","Q 2244 3097 1772 3097 \n","Q 1472 3097 1187 3025 \n","Q 903 2953 641 2809 \n","L 641 3341 \n","Q 956 3463 1253 3523 \n","Q 1550 3584 1831 3584 \n","Q 2591 3584 2966 3190 \n","Q 3341 2797 3341 1997 \n","z\n","\" transform=\"scale(0.015625)\"/>\n","       <path id=\"DejaVuSans-69\" d=\"M 603 3500 \n","L 1178 3500 \n","L 1178 0 \n","L 603 0 \n","L 603 3500 \n","z\n","M 603 4863 \n","L 1178 4863 \n","L 1178 4134 \n","L 603 4134 \n","L 603 4863 \n","z\n","\" transform=\"scale(0.015625)\"/>\n","       <path id=\"DejaVuSans-6e\" d=\"M 3513 2113 \n","L 3513 0 \n","L 2938 0 \n","L 2938 2094 \n","Q 2938 2591 2744 2837 \n","Q 2550 3084 2163 3084 \n","Q 1697 3084 1428 2787 \n","Q 1159 2491 1159 1978 \n","L 1159 0 \n","L 581 0 \n","L 581 3500 \n","L 1159 3500 \n","L 1159 2956 \n","Q 1366 3272 1645 3428 \n","Q 1925 3584 2291 3584 \n","Q 2894 3584 3203 3211 \n","Q 3513 2838 3513 2113 \n","z\n","\" transform=\"scale(0.015625)\"/>\n","       <path id=\"DejaVuSans-20\" transform=\"scale(0.015625)\"/>\n","       <path id=\"DejaVuSans-6c\" d=\"M 603 4863 \n","L 1178 4863 \n","L 1178 0 \n","L 603 0 \n","L 603 4863 \n","z\n","\" transform=\"scale(0.015625)\"/>\n","       <path id=\"DejaVuSans-73\" d=\"M 2834 3397 \n","L 2834 2853 \n","Q 2591 2978 2328 3040 \n","Q 2066 3103 1784 3103 \n","Q 1356 3103 1142 2972 \n","Q 928 2841 928 2578 \n","Q 928 2378 1081 2264 \n","Q 1234 2150 1697 2047 \n","L 1894 2003 \n","Q 2506 1872 2764 1633 \n","Q 3022 1394 3022 966 \n","Q 3022 478 2636 193 \n","Q 2250 -91 1575 -91 \n","Q 1294 -91 989 -36 \n","Q 684 19 347 128 \n","L 347 722 \n","Q 666 556 975 473 \n","Q 1284 391 1588 391 \n","Q 1994 391 2212 530 \n","Q 2431 669 2431 922 \n","Q 2431 1156 2273 1281 \n","Q 2116 1406 1581 1522 \n","L 1381 1569 \n","Q 847 1681 609 1914 \n","Q 372 2147 372 2553 \n","Q 372 3047 722 3315 \n","Q 1072 3584 1716 3584 \n","Q 2034 3584 2315 3537 \n","Q 2597 3491 2834 3397 \n","z\n","\" transform=\"scale(0.015625)\"/>\n","      </defs>\n","      <use xlink:href=\"#DejaVuSans-74\"/>\n","      <use xlink:href=\"#DejaVuSans-72\" x=\"39.208984\"/>\n","      <use xlink:href=\"#DejaVuSans-61\" x=\"80.322266\"/>\n","      <use xlink:href=\"#DejaVuSans-69\" x=\"141.601562\"/>\n","      <use xlink:href=\"#DejaVuSans-6e\" x=\"169.384766\"/>\n","      <use xlink:href=\"#DejaVuSans-20\" x=\"232.763672\"/>\n","      <use xlink:href=\"#DejaVuSans-6c\" x=\"264.550781\"/>\n","      <use xlink:href=\"#DejaVuSans-6f\" x=\"292.333984\"/>\n","      <use xlink:href=\"#DejaVuSans-73\" x=\"353.515625\"/>\n","      <use xlink:href=\"#DejaVuSans-73\" x=\"405.615234\"/>\n","     </g>\n","    </g>\n","    <g id=\"line2d_21\">\n","     <path d=\"M 142.634375 74.409375 \n","L 152.634375 74.409375 \n","L 162.634375 74.409375 \n","\" style=\"fill: none; stroke-dasharray: 5.55,2.4; stroke-dashoffset: 0; stroke: #bf00bf; stroke-width: 1.5\"/>\n","    </g>\n","    <g id=\"text_11\">\n","     <!-- train acc -->\n","     <g transform=\"translate(170.634375 77.909375)scale(0.1 -0.1)\">\n","      <use xlink:href=\"#DejaVuSans-74\"/>\n","      <use xlink:href=\"#DejaVuSans-72\" x=\"39.208984\"/>\n","      <use xlink:href=\"#DejaVuSans-61\" x=\"80.322266\"/>\n","      <use xlink:href=\"#DejaVuSans-69\" x=\"141.601562\"/>\n","      <use xlink:href=\"#DejaVuSans-6e\" x=\"169.384766\"/>\n","      <use xlink:href=\"#DejaVuSans-20\" x=\"232.763672\"/>\n","      <use xlink:href=\"#DejaVuSans-61\" x=\"264.550781\"/>\n","      <use xlink:href=\"#DejaVuSans-63\" x=\"325.830078\"/>\n","      <use xlink:href=\"#DejaVuSans-63\" x=\"380.810547\"/>\n","     </g>\n","    </g>\n","    <g id=\"line2d_22\">\n","     <path d=\"M 142.634375 89.0875 \n","L 152.634375 89.0875 \n","L 162.634375 89.0875 \n","\" style=\"fill: none; stroke-dasharray: 9.6,2.4,1.5,2.4; stroke-dashoffset: 0; stroke: #008000; stroke-width: 1.5\"/>\n","    </g>\n","    <g id=\"text_12\">\n","     <!-- test acc -->\n","     <g transform=\"translate(170.634375 92.5875)scale(0.1 -0.1)\">\n","      <use xlink:href=\"#DejaVuSans-74\"/>\n","      <use xlink:href=\"#DejaVuSans-65\" x=\"39.208984\"/>\n","      <use xlink:href=\"#DejaVuSans-73\" x=\"100.732422\"/>\n","      <use xlink:href=\"#DejaVuSans-74\" x=\"152.832031\"/>\n","      <use xlink:href=\"#DejaVuSans-20\" x=\"192.041016\"/>\n","      <use xlink:href=\"#DejaVuSans-61\" x=\"223.828125\"/>\n","      <use xlink:href=\"#DejaVuSans-63\" x=\"285.107422\"/>\n","      <use xlink:href=\"#DejaVuSans-63\" x=\"340.087891\"/>\n","     </g>\n","    </g>\n","   </g>\n","  </g>\n"," </g>\n"," <defs>\n","  <clipPath id=\"p1f1a59b8d9\">\n","   <rect x=\"30.103125\" y=\"7.2\" width=\"195.3\" height=\"135.9\"/>\n","  </clipPath>\n"," </defs>\n","</svg>\n"],"text/plain":["<Figure size 252x180 with 1 Axes>"]},"metadata":{"needs_background":"light"},"output_type":"display_data"}],"source":["train_iter, test_iter = d2l.load_data_fashion_mnist(batch_size)\n","d2l.train_ch3(net, train_iter, test_iter, loss, num_epochs, trainer)"],"id":"cfef131f"},{"cell_type":"markdown","metadata":{"origin_pos":13,"id":"8bfa861c"},"source":["## 小结\n","\n","* 我们可以使用高级API更简洁地实现多层感知机。\n","* 对于相同的分类问题，多层感知机的实现与softmax回归的实现相同，只是多层感知机的实现里增加了带有激活函数的隐藏层。\n","\n","## 练习\n","\n","1. 尝试添加不同数量的隐藏层（也可以修改学习率），怎么样设置效果最好？\n","1. 尝试不同的激活函数，哪个效果最好？\n","1. 尝试不同的方案来初始化权重，什么方法效果最好？\n"],"id":"8bfa861c"},{"cell_type":"markdown","source":[],"metadata":{"id":"PFhNdZB40ikP"},"id":"PFhNdZB40ikP"},{"cell_type":"markdown","metadata":{"origin_pos":0,"id":"OOagwkyR0i97"},"source":["# 多层感知机的简洁实现\n",":label:`sec_mlp_concise`\n","\n","正如你所期待的，我们可以(**通过高级API更简洁地实现多层感知机**)。\n"],"id":"OOagwkyR0i97"},{"cell_type":"code","execution_count":null,"metadata":{"execution":{"iopub.execute_input":"2022-07-31T02:29:21.553674Z","iopub.status.busy":"2022-07-31T02:29:21.553096Z","iopub.status.idle":"2022-07-31T02:29:23.419894Z","shell.execute_reply":"2022-07-31T02:29:23.419131Z"},"origin_pos":2,"tab":["pytorch"],"id":"01ypn3pw0i98"},"outputs":[],"source":["import torch\n","from torch import nn\n","from d2l import torch as d2l"],"id":"01ypn3pw0i98"},{"cell_type":"markdown","metadata":{"origin_pos":4,"id":"GsGaKnjv0i98"},"source":["## 模型\n","\n","与softmax回归的简洁实现（ :numref:`sec_softmax_concise`）相比，\n","唯一的区别是我们添加了2个全连接层（之前我们只添加了1个全连接层）。\n","第一层是[**隐藏层**]，它(**包含256个隐藏单元，并使用了ReLU激活函数**)。\n","第二层是输出层。\n"],"id":"GsGaKnjv0i98"},{"cell_type":"code","execution_count":null,"metadata":{"execution":{"iopub.execute_input":"2022-07-31T02:29:23.424678Z","iopub.status.busy":"2022-07-31T02:29:23.424361Z","iopub.status.idle":"2022-07-31T02:29:23.433077Z","shell.execute_reply":"2022-07-31T02:29:23.432448Z"},"origin_pos":6,"tab":["pytorch"],"id":"qnoL461c0i98"},"outputs":[],"source":["net = nn.Sequential(nn.Flatten(),\n","                    nn.Linear(784, 256),\n","                    nn.ReLU(),\n","                    nn.Linear(256, 10))\n","\n","def init_weights(m):\n","    if type(m) == nn.Linear:\n","        nn.init.normal_(m.weight, std=0.01)\n","\n","net.apply(init_weights);"],"id":"qnoL461c0i98"},{"cell_type":"markdown","metadata":{"origin_pos":8,"id":"dsYUtqvt0i98"},"source":["[**训练过程**]的实现与我们实现softmax回归时完全相同，\n","这种模块化设计使我们能够将与模型架构有关的内容独立出来。\n"],"id":"dsYUtqvt0i98"},{"cell_type":"code","execution_count":null,"metadata":{"execution":{"iopub.execute_input":"2022-07-31T02:29:23.437358Z","iopub.status.busy":"2022-07-31T02:29:23.436999Z","iopub.status.idle":"2022-07-31T02:29:23.440994Z","shell.execute_reply":"2022-07-31T02:29:23.440346Z"},"origin_pos":10,"tab":["pytorch"],"id":"EIW8NsHH0i98"},"outputs":[],"source":["batch_size, lr, num_epochs = 256, 0.1, 10\n","loss = nn.CrossEntropyLoss(reduction='none')\n","trainer = torch.optim.SGD(net.parameters(), lr=lr)"],"id":"EIW8NsHH0i98"},{"cell_type":"code","execution_count":null,"metadata":{"execution":{"iopub.execute_input":"2022-07-31T02:29:23.445300Z","iopub.status.busy":"2022-07-31T02:29:23.444959Z","iopub.status.idle":"2022-07-31T02:29:57.656280Z","shell.execute_reply":"2022-07-31T02:29:57.655475Z"},"origin_pos":12,"tab":["pytorch"],"outputId":"0855b85e-a28a-4894-a337-80c1ab80ea64","id":"UYsrUQA20i98"},"outputs":[{"data":{"image/svg+xml":["<?xml version=\"1.0\" encoding=\"utf-8\" standalone=\"no\"?>\n","<!DOCTYPE svg PUBLIC \"-//W3C//DTD SVG 1.1//EN\"\n","  \"http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd\">\n","<svg xmlns:xlink=\"http://www.w3.org/1999/xlink\" width=\"238.965625pt\" height=\"180.65625pt\" viewBox=\"0 0 238.965625 180.65625\" xmlns=\"http://www.w3.org/2000/svg\" version=\"1.1\">\n"," <metadata>\n","  <rdf:RDF xmlns:dc=\"http://purl.org/dc/elements/1.1/\" xmlns:cc=\"http://creativecommons.org/ns#\" xmlns:rdf=\"http://www.w3.org/1999/02/22-rdf-syntax-ns#\">\n","   <cc:Work>\n","    <dc:type rdf:resource=\"http://purl.org/dc/dcmitype/StillImage\"/>\n","    <dc:date>2022-07-31T02:29:57.616205</dc:date>\n","    <dc:format>image/svg+xml</dc:format>\n","    <dc:creator>\n","     <cc:Agent>\n","      <dc:title>Matplotlib v3.5.1, https://matplotlib.org/</dc:title>\n","     </cc:Agent>\n","    </dc:creator>\n","   </cc:Work>\n","  </rdf:RDF>\n"," </metadata>\n"," <defs>\n","  <style type=\"text/css\">*{stroke-linejoin: round; stroke-linecap: butt}</style>\n"," </defs>\n"," <g id=\"figure_1\">\n","  <g id=\"patch_1\">\n","   <path d=\"M 0 180.65625 \n","L 238.965625 180.65625 \n","L 238.965625 0 \n","L 0 0 \n","L 0 180.65625 \n","z\n","\" style=\"fill: none\"/>\n","  </g>\n","  <g id=\"axes_1\">\n","   <g id=\"patch_2\">\n","    <path d=\"M 30.103125 143.1 \n","L 225.403125 143.1 \n","L 225.403125 7.2 \n","L 30.103125 7.2 \n","z\n","\" style=\"fill: #ffffff\"/>\n","   </g>\n","   <g id=\"matplotlib.axis_1\">\n","    <g id=\"xtick_1\">\n","     <g id=\"line2d_1\">\n","      <path d=\"M 51.803125 143.1 \n","L 51.803125 7.2 \n","\" clip-path=\"url(#p1f1a59b8d9)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n","     </g>\n","     <g id=\"line2d_2\">\n","      <defs>\n","       <path id=\"m87f692bb68\" d=\"M 0 0 \n","L 0 3.5 \n","\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n","      </defs>\n","      <g>\n","       <use xlink:href=\"#m87f692bb68\" x=\"51.803125\" y=\"143.1\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n","      </g>\n","     </g>\n","     <g id=\"text_1\">\n","      <!-- 2 -->\n","      <g transform=\"translate(48.621875 157.698438)scale(0.1 -0.1)\">\n","       <defs>\n","        <path id=\"DejaVuSans-32\" d=\"M 1228 531 \n","L 3431 531 \n","L 3431 0 \n","L 469 0 \n","L 469 531 \n","Q 828 903 1448 1529 \n","Q 2069 2156 2228 2338 \n","Q 2531 2678 2651 2914 \n","Q 2772 3150 2772 3378 \n","Q 2772 3750 2511 3984 \n","Q 2250 4219 1831 4219 \n","Q 1534 4219 1204 4116 \n","Q 875 4013 500 3803 \n","L 500 4441 \n","Q 881 4594 1212 4672 \n","Q 1544 4750 1819 4750 \n","Q 2544 4750 2975 4387 \n","Q 3406 4025 3406 3419 \n","Q 3406 3131 3298 2873 \n","Q 3191 2616 2906 2266 \n","Q 2828 2175 2409 1742 \n","Q 1991 1309 1228 531 \n","z\n","\" transform=\"scale(0.015625)\"/>\n","       </defs>\n","       <use xlink:href=\"#DejaVuSans-32\"/>\n","      </g>\n","     </g>\n","    </g>\n","    <g id=\"xtick_2\">\n","     <g id=\"line2d_3\">\n","      <path d=\"M 95.203125 143.1 \n","L 95.203125 7.2 \n","\" clip-path=\"url(#p1f1a59b8d9)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n","     </g>\n","     <g id=\"line2d_4\">\n","      <g>\n","       <use xlink:href=\"#m87f692bb68\" x=\"95.203125\" y=\"143.1\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n","      </g>\n","     </g>\n","     <g id=\"text_2\">\n","      <!-- 4 -->\n","      <g transform=\"translate(92.021875 157.698438)scale(0.1 -0.1)\">\n","       <defs>\n","        <path id=\"DejaVuSans-34\" d=\"M 2419 4116 \n","L 825 1625 \n","L 2419 1625 \n","L 2419 4116 \n","z\n","M 2253 4666 \n","L 3047 4666 \n","L 3047 1625 \n","L 3713 1625 \n","L 3713 1100 \n","L 3047 1100 \n","L 3047 0 \n","L 2419 0 \n","L 2419 1100 \n","L 313 1100 \n","L 313 1709 \n","L 2253 4666 \n","z\n","\" transform=\"scale(0.015625)\"/>\n","       </defs>\n","       <use xlink:href=\"#DejaVuSans-34\"/>\n","      </g>\n","     </g>\n","    </g>\n","    <g id=\"xtick_3\">\n","     <g id=\"line2d_5\">\n","      <path d=\"M 138.603125 143.1 \n","L 138.603125 7.2 \n","\" clip-path=\"url(#p1f1a59b8d9)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n","     </g>\n","     <g id=\"line2d_6\">\n","      <g>\n","       <use xlink:href=\"#m87f692bb68\" x=\"138.603125\" y=\"143.1\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n","      </g>\n","     </g>\n","     <g id=\"text_3\">\n","      <!-- 6 -->\n","      <g transform=\"translate(135.421875 157.698438)scale(0.1 -0.1)\">\n","       <defs>\n","        <path id=\"DejaVuSans-36\" d=\"M 2113 2584 \n","Q 1688 2584 1439 2293 \n","Q 1191 2003 1191 1497 \n","Q 1191 994 1439 701 \n","Q 1688 409 2113 409 \n","Q 2538 409 2786 701 \n","Q 3034 994 3034 1497 \n","Q 3034 2003 2786 2293 \n","Q 2538 2584 2113 2584 \n","z\n","M 3366 4563 \n","L 3366 3988 \n","Q 3128 4100 2886 4159 \n","Q 2644 4219 2406 4219 \n","Q 1781 4219 1451 3797 \n","Q 1122 3375 1075 2522 \n","Q 1259 2794 1537 2939 \n","Q 1816 3084 2150 3084 \n","Q 2853 3084 3261 2657 \n","Q 3669 2231 3669 1497 \n","Q 3669 778 3244 343 \n","Q 2819 -91 2113 -91 \n","Q 1303 -91 875 529 \n","Q 447 1150 447 2328 \n","Q 447 3434 972 4092 \n","Q 1497 4750 2381 4750 \n","Q 2619 4750 2861 4703 \n","Q 3103 4656 3366 4563 \n","z\n","\" transform=\"scale(0.015625)\"/>\n","       </defs>\n","       <use xlink:href=\"#DejaVuSans-36\"/>\n","      </g>\n","     </g>\n","    </g>\n","    <g id=\"xtick_4\">\n","     <g id=\"line2d_7\">\n","      <path d=\"M 182.003125 143.1 \n","L 182.003125 7.2 \n","\" clip-path=\"url(#p1f1a59b8d9)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n","     </g>\n","     <g id=\"line2d_8\">\n","      <g>\n","       <use xlink:href=\"#m87f692bb68\" x=\"182.003125\" y=\"143.1\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n","      </g>\n","     </g>\n","     <g id=\"text_4\">\n","      <!-- 8 -->\n","      <g transform=\"translate(178.821875 157.698438)scale(0.1 -0.1)\">\n","       <defs>\n","        <path id=\"DejaVuSans-38\" d=\"M 2034 2216 \n","Q 1584 2216 1326 1975 \n","Q 1069 1734 1069 1313 \n","Q 1069 891 1326 650 \n","Q 1584 409 2034 409 \n","Q 2484 409 2743 651 \n","Q 3003 894 3003 1313 \n","Q 3003 1734 2745 1975 \n","Q 2488 2216 2034 2216 \n","z\n","M 1403 2484 \n","Q 997 2584 770 2862 \n","Q 544 3141 544 3541 \n","Q 544 4100 942 4425 \n","Q 1341 4750 2034 4750 \n","Q 2731 4750 3128 4425 \n","Q 3525 4100 3525 3541 \n","Q 3525 3141 3298 2862 \n","Q 3072 2584 2669 2484 \n","Q 3125 2378 3379 2068 \n","Q 3634 1759 3634 1313 \n","Q 3634 634 3220 271 \n","Q 2806 -91 2034 -91 \n","Q 1263 -91 848 271 \n","Q 434 634 434 1313 \n","Q 434 1759 690 2068 \n","Q 947 2378 1403 2484 \n","z\n","M 1172 3481 \n","Q 1172 3119 1398 2916 \n","Q 1625 2713 2034 2713 \n","Q 2441 2713 2670 2916 \n","Q 2900 3119 2900 3481 \n","Q 2900 3844 2670 4047 \n","Q 2441 4250 2034 4250 \n","Q 1625 4250 1398 4047 \n","Q 1172 3844 1172 3481 \n","z\n","\" transform=\"scale(0.015625)\"/>\n","       </defs>\n","       <use xlink:href=\"#DejaVuSans-38\"/>\n","      </g>\n","     </g>\n","    </g>\n","    <g id=\"xtick_5\">\n","     <g id=\"line2d_9\">\n","      <path d=\"M 225.403125 143.1 \n","L 225.403125 7.2 \n","\" clip-path=\"url(#p1f1a59b8d9)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n","     </g>\n","     <g id=\"line2d_10\">\n","      <g>\n","       <use xlink:href=\"#m87f692bb68\" x=\"225.403125\" y=\"143.1\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n","      </g>\n","     </g>\n","     <g id=\"text_5\">\n","      <!-- 10 -->\n","      <g transform=\"translate(219.040625 157.698438)scale(0.1 -0.1)\">\n","       <defs>\n","        <path id=\"DejaVuSans-31\" d=\"M 794 531 \n","L 1825 531 \n","L 1825 4091 \n","L 703 3866 \n","L 703 4441 \n","L 1819 4666 \n","L 2450 4666 \n","L 2450 531 \n","L 3481 531 \n","L 3481 0 \n","L 794 0 \n","L 794 531 \n","z\n","\" transform=\"scale(0.015625)\"/>\n","        <path id=\"DejaVuSans-30\" d=\"M 2034 4250 \n","Q 1547 4250 1301 3770 \n","Q 1056 3291 1056 2328 \n","Q 1056 1369 1301 889 \n","Q 1547 409 2034 409 \n","Q 2525 409 2770 889 \n","Q 3016 1369 3016 2328 \n","Q 3016 3291 2770 3770 \n","Q 2525 4250 2034 4250 \n","z\n","M 2034 4750 \n","Q 2819 4750 3233 4129 \n","Q 3647 3509 3647 2328 \n","Q 3647 1150 3233 529 \n","Q 2819 -91 2034 -91 \n","Q 1250 -91 836 529 \n","Q 422 1150 422 2328 \n","Q 422 3509 836 4129 \n","Q 1250 4750 2034 4750 \n","z\n","\" transform=\"scale(0.015625)\"/>\n","       </defs>\n","       <use xlink:href=\"#DejaVuSans-31\"/>\n","       <use xlink:href=\"#DejaVuSans-30\" x=\"63.623047\"/>\n","      </g>\n","     </g>\n","    </g>\n","    <g id=\"text_6\">\n","     <!-- epoch -->\n","     <g transform=\"translate(112.525 171.376563)scale(0.1 -0.1)\">\n","      <defs>\n","       <path id=\"DejaVuSans-65\" d=\"M 3597 1894 \n","L 3597 1613 \n","L 953 1613 \n","Q 991 1019 1311 708 \n","Q 1631 397 2203 397 \n","Q 2534 397 2845 478 \n","Q 3156 559 3463 722 \n","L 3463 178 \n","Q 3153 47 2828 -22 \n","Q 2503 -91 2169 -91 \n","Q 1331 -91 842 396 \n","Q 353 884 353 1716 \n","Q 353 2575 817 3079 \n","Q 1281 3584 2069 3584 \n","Q 2775 3584 3186 3129 \n","Q 3597 2675 3597 1894 \n","z\n","M 3022 2063 \n","Q 3016 2534 2758 2815 \n","Q 2500 3097 2075 3097 \n","Q 1594 3097 1305 2825 \n","Q 1016 2553 972 2059 \n","L 3022 2063 \n","z\n","\" transform=\"scale(0.015625)\"/>\n","       <path id=\"DejaVuSans-70\" d=\"M 1159 525 \n","L 1159 -1331 \n","L 581 -1331 \n","L 581 3500 \n","L 1159 3500 \n","L 1159 2969 \n","Q 1341 3281 1617 3432 \n","Q 1894 3584 2278 3584 \n","Q 2916 3584 3314 3078 \n","Q 3713 2572 3713 1747 \n","Q 3713 922 3314 415 \n","Q 2916 -91 2278 -91 \n","Q 1894 -91 1617 61 \n","Q 1341 213 1159 525 \n","z\n","M 3116 1747 \n","Q 3116 2381 2855 2742 \n","Q 2594 3103 2138 3103 \n","Q 1681 3103 1420 2742 \n","Q 1159 2381 1159 1747 \n","Q 1159 1113 1420 752 \n","Q 1681 391 2138 391 \n","Q 2594 391 2855 752 \n","Q 3116 1113 3116 1747 \n","z\n","\" transform=\"scale(0.015625)\"/>\n","       <path id=\"DejaVuSans-6f\" d=\"M 1959 3097 \n","Q 1497 3097 1228 2736 \n","Q 959 2375 959 1747 \n","Q 959 1119 1226 758 \n","Q 1494 397 1959 397 \n","Q 2419 397 2687 759 \n","Q 2956 1122 2956 1747 \n","Q 2956 2369 2687 2733 \n","Q 2419 3097 1959 3097 \n","z\n","M 1959 3584 \n","Q 2709 3584 3137 3096 \n","Q 3566 2609 3566 1747 \n","Q 3566 888 3137 398 \n","Q 2709 -91 1959 -91 \n","Q 1206 -91 779 398 \n","Q 353 888 353 1747 \n","Q 353 2609 779 3096 \n","Q 1206 3584 1959 3584 \n","z\n","\" transform=\"scale(0.015625)\"/>\n","       <path id=\"DejaVuSans-63\" d=\"M 3122 3366 \n","L 3122 2828 \n","Q 2878 2963 2633 3030 \n","Q 2388 3097 2138 3097 \n","Q 1578 3097 1268 2742 \n","Q 959 2388 959 1747 \n","Q 959 1106 1268 751 \n","Q 1578 397 2138 397 \n","Q 2388 397 2633 464 \n","Q 2878 531 3122 666 \n","L 3122 134 \n","Q 2881 22 2623 -34 \n","Q 2366 -91 2075 -91 \n","Q 1284 -91 818 406 \n","Q 353 903 353 1747 \n","Q 353 2603 823 3093 \n","Q 1294 3584 2113 3584 \n","Q 2378 3584 2631 3529 \n","Q 2884 3475 3122 3366 \n","z\n","\" transform=\"scale(0.015625)\"/>\n","       <path id=\"DejaVuSans-68\" d=\"M 3513 2113 \n","L 3513 0 \n","L 2938 0 \n","L 2938 2094 \n","Q 2938 2591 2744 2837 \n","Q 2550 3084 2163 3084 \n","Q 1697 3084 1428 2787 \n","Q 1159 2491 1159 1978 \n","L 1159 0 \n","L 581 0 \n","L 581 4863 \n","L 1159 4863 \n","L 1159 2956 \n","Q 1366 3272 1645 3428 \n","Q 1925 3584 2291 3584 \n","Q 2894 3584 3203 3211 \n","Q 3513 2838 3513 2113 \n","z\n","\" transform=\"scale(0.015625)\"/>\n","      </defs>\n","      <use xlink:href=\"#DejaVuSans-65\"/>\n","      <use xlink:href=\"#DejaVuSans-70\" x=\"61.523438\"/>\n","      <use xlink:href=\"#DejaVuSans-6f\" x=\"125\"/>\n","      <use xlink:href=\"#DejaVuSans-63\" x=\"186.181641\"/>\n","      <use xlink:href=\"#DejaVuSans-68\" x=\"241.162109\"/>\n","     </g>\n","    </g>\n","   </g>\n","   <g id=\"matplotlib.axis_2\">\n","    <g id=\"ytick_1\">\n","     <g id=\"line2d_11\">\n","      <path d=\"M 30.103125 120.45 \n","L 225.403125 120.45 \n","\" clip-path=\"url(#p1f1a59b8d9)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n","     </g>\n","     <g id=\"line2d_12\">\n","      <defs>\n","       <path id=\"mfbf14b93b2\" d=\"M 0 0 \n","L -3.5 0 \n","\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n","      </defs>\n","      <g>\n","       <use xlink:href=\"#mfbf14b93b2\" x=\"30.103125\" y=\"120.45\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n","      </g>\n","     </g>\n","     <g id=\"text_7\">\n","      <!-- 0.4 -->\n","      <g transform=\"translate(7.2 124.249219)scale(0.1 -0.1)\">\n","       <defs>\n","        <path id=\"DejaVuSans-2e\" d=\"M 684 794 \n","L 1344 794 \n","L 1344 0 \n","L 684 0 \n","L 684 794 \n","z\n","\" transform=\"scale(0.015625)\"/>\n","       </defs>\n","       <use xlink:href=\"#DejaVuSans-30\"/>\n","       <use xlink:href=\"#DejaVuSans-2e\" x=\"63.623047\"/>\n","       <use xlink:href=\"#DejaVuSans-34\" x=\"95.410156\"/>\n","      </g>\n","     </g>\n","    </g>\n","    <g id=\"ytick_2\">\n","     <g id=\"line2d_13\">\n","      <path d=\"M 30.103125 75.15 \n","L 225.403125 75.15 \n","\" clip-path=\"url(#p1f1a59b8d9)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n","     </g>\n","     <g id=\"line2d_14\">\n","      <g>\n","       <use xlink:href=\"#mfbf14b93b2\" x=\"30.103125\" y=\"75.15\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n","      </g>\n","     </g>\n","     <g id=\"text_8\">\n","      <!-- 0.6 -->\n","      <g transform=\"translate(7.2 78.949219)scale(0.1 -0.1)\">\n","       <use xlink:href=\"#DejaVuSans-30\"/>\n","       <use xlink:href=\"#DejaVuSans-2e\" x=\"63.623047\"/>\n","       <use xlink:href=\"#DejaVuSans-36\" x=\"95.410156\"/>\n","      </g>\n","     </g>\n","    </g>\n","    <g id=\"ytick_3\">\n","     <g id=\"line2d_15\">\n","      <path d=\"M 30.103125 29.85 \n","L 225.403125 29.85 \n","\" clip-path=\"url(#p1f1a59b8d9)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n","     </g>\n","     <g id=\"line2d_16\">\n","      <g>\n","       <use xlink:href=\"#mfbf14b93b2\" x=\"30.103125\" y=\"29.85\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n","      </g>\n","     </g>\n","     <g id=\"text_9\">\n","      <!-- 0.8 -->\n","      <g transform=\"translate(7.2 33.649219)scale(0.1 -0.1)\">\n","       <use xlink:href=\"#DejaVuSans-30\"/>\n","       <use xlink:href=\"#DejaVuSans-2e\" x=\"63.623047\"/>\n","       <use xlink:href=\"#DejaVuSans-38\" x=\"95.410156\"/>\n","      </g>\n","     </g>\n","    </g>\n","   </g>\n","   <g id=\"line2d_17\">\n","    <path d=\"M 35.085071 -1 \n","L 51.803125 76.810298 \n","L 73.503125 93.142551 \n","L 95.203125 102.471887 \n","L 116.903125 108.512262 \n","L 138.603125 112.745328 \n","L 160.303125 116.733151 \n","L 182.003125 118.982144 \n","L 203.703125 121.986568 \n","L 225.403125 124.066666 \n","\" clip-path=\"url(#p1f1a59b8d9)\" style=\"fill: none; stroke: #1f77b4; stroke-width: 1.5; stroke-linecap: square\"/>\n","   </g>\n","   <g id=\"line2d_18\">\n","    <path d=\"M 30.103125 66.697775 \n","L 51.803125 31.439275 \n","L 73.503125 25.88625 \n","L 95.203125 22.507625 \n","L 116.903125 20.476675 \n","L 138.603125 19.25735 \n","L 160.303125 17.720925 \n","L 182.003125 16.84135 \n","L 203.703125 15.991975 \n","L 225.403125 15.297375 \n","\" clip-path=\"url(#p1f1a59b8d9)\" style=\"fill: none; stroke-dasharray: 5.55,2.4; stroke-dashoffset: 0; stroke: #bf00bf; stroke-width: 1.5\"/>\n","   </g>\n","   <g id=\"line2d_19\">\n","    <path d=\"M 30.103125 51.6393 \n","L 51.803125 29.82735 \n","L 73.503125 27.97005 \n","L 95.203125 32.8398 \n","L 116.903125 30.303 \n","L 138.603125 21.3789 \n","L 160.303125 22.1037 \n","L 182.003125 19.9293 \n","L 203.703125 18.7062 \n","L 225.403125 19.90665 \n","\" clip-path=\"url(#p1f1a59b8d9)\" style=\"fill: none; stroke-dasharray: 9.6,2.4,1.5,2.4; stroke-dashoffset: 0; stroke: #008000; stroke-width: 1.5\"/>\n","   </g>\n","   <g id=\"patch_3\">\n","    <path d=\"M 30.103125 143.1 \n","L 30.103125 7.2 \n","\" style=\"fill: none; stroke: #000000; stroke-width: 0.8; stroke-linejoin: miter; stroke-linecap: square\"/>\n","   </g>\n","   <g id=\"patch_4\">\n","    <path d=\"M 225.403125 143.1 \n","L 225.403125 7.2 \n","\" style=\"fill: none; stroke: #000000; stroke-width: 0.8; stroke-linejoin: miter; stroke-linecap: square\"/>\n","   </g>\n","   <g id=\"patch_5\">\n","    <path d=\"M 30.103125 143.1 \n","L 225.403125 143.1 \n","\" style=\"fill: none; stroke: #000000; stroke-width: 0.8; stroke-linejoin: miter; stroke-linecap: square\"/>\n","   </g>\n","   <g id=\"patch_6\">\n","    <path d=\"M 30.103125 7.2 \n","L 225.403125 7.2 \n","\" style=\"fill: none; stroke: #000000; stroke-width: 0.8; stroke-linejoin: miter; stroke-linecap: square\"/>\n","   </g>\n","   <g id=\"legend_1\">\n","    <g id=\"patch_7\">\n","     <path d=\"M 140.634375 98.667187 \n","L 218.403125 98.667187 \n","Q 220.403125 98.667187 220.403125 96.667187 \n","L 220.403125 53.632812 \n","Q 220.403125 51.632812 218.403125 51.632812 \n","L 140.634375 51.632812 \n","Q 138.634375 51.632812 138.634375 53.632812 \n","L 138.634375 96.667187 \n","Q 138.634375 98.667187 140.634375 98.667187 \n","z\n","\" style=\"fill: #ffffff; opacity: 0.8; stroke: #cccccc; stroke-linejoin: miter\"/>\n","    </g>\n","    <g id=\"line2d_20\">\n","     <path d=\"M 142.634375 59.73125 \n","L 152.634375 59.73125 \n","L 162.634375 59.73125 \n","\" style=\"fill: none; stroke: #1f77b4; stroke-width: 1.5; stroke-linecap: square\"/>\n","    </g>\n","    <g id=\"text_10\">\n","     <!-- train loss -->\n","     <g transform=\"translate(170.634375 63.23125)scale(0.1 -0.1)\">\n","      <defs>\n","       <path id=\"DejaVuSans-74\" d=\"M 1172 4494 \n","L 1172 3500 \n","L 2356 3500 \n","L 2356 3053 \n","L 1172 3053 \n","L 1172 1153 \n","Q 1172 725 1289 603 \n","Q 1406 481 1766 481 \n","L 2356 481 \n","L 2356 0 \n","L 1766 0 \n","Q 1100 0 847 248 \n","Q 594 497 594 1153 \n","L 594 3053 \n","L 172 3053 \n","L 172 3500 \n","L 594 3500 \n","L 594 4494 \n","L 1172 4494 \n","z\n","\" transform=\"scale(0.015625)\"/>\n","       <path id=\"DejaVuSans-72\" d=\"M 2631 2963 \n","Q 2534 3019 2420 3045 \n","Q 2306 3072 2169 3072 \n","Q 1681 3072 1420 2755 \n","Q 1159 2438 1159 1844 \n","L 1159 0 \n","L 581 0 \n","L 581 3500 \n","L 1159 3500 \n","L 1159 2956 \n","Q 1341 3275 1631 3429 \n","Q 1922 3584 2338 3584 \n","Q 2397 3584 2469 3576 \n","Q 2541 3569 2628 3553 \n","L 2631 2963 \n","z\n","\" transform=\"scale(0.015625)\"/>\n","       <path id=\"DejaVuSans-61\" d=\"M 2194 1759 \n","Q 1497 1759 1228 1600 \n","Q 959 1441 959 1056 \n","Q 959 750 1161 570 \n","Q 1363 391 1709 391 \n","Q 2188 391 2477 730 \n","Q 2766 1069 2766 1631 \n","L 2766 1759 \n","L 2194 1759 \n","z\n","M 3341 1997 \n","L 3341 0 \n","L 2766 0 \n","L 2766 531 \n","Q 2569 213 2275 61 \n","Q 1981 -91 1556 -91 \n","Q 1019 -91 701 211 \n","Q 384 513 384 1019 \n","Q 384 1609 779 1909 \n","Q 1175 2209 1959 2209 \n","L 2766 2209 \n","L 2766 2266 \n","Q 2766 2663 2505 2880 \n","Q 2244 3097 1772 3097 \n","Q 1472 3097 1187 3025 \n","Q 903 2953 641 2809 \n","L 641 3341 \n","Q 956 3463 1253 3523 \n","Q 1550 3584 1831 3584 \n","Q 2591 3584 2966 3190 \n","Q 3341 2797 3341 1997 \n","z\n","\" transform=\"scale(0.015625)\"/>\n","       <path id=\"DejaVuSans-69\" d=\"M 603 3500 \n","L 1178 3500 \n","L 1178 0 \n","L 603 0 \n","L 603 3500 \n","z\n","M 603 4863 \n","L 1178 4863 \n","L 1178 4134 \n","L 603 4134 \n","L 603 4863 \n","z\n","\" transform=\"scale(0.015625)\"/>\n","       <path id=\"DejaVuSans-6e\" d=\"M 3513 2113 \n","L 3513 0 \n","L 2938 0 \n","L 2938 2094 \n","Q 2938 2591 2744 2837 \n","Q 2550 3084 2163 3084 \n","Q 1697 3084 1428 2787 \n","Q 1159 2491 1159 1978 \n","L 1159 0 \n","L 581 0 \n","L 581 3500 \n","L 1159 3500 \n","L 1159 2956 \n","Q 1366 3272 1645 3428 \n","Q 1925 3584 2291 3584 \n","Q 2894 3584 3203 3211 \n","Q 3513 2838 3513 2113 \n","z\n","\" transform=\"scale(0.015625)\"/>\n","       <path id=\"DejaVuSans-20\" transform=\"scale(0.015625)\"/>\n","       <path id=\"DejaVuSans-6c\" d=\"M 603 4863 \n","L 1178 4863 \n","L 1178 0 \n","L 603 0 \n","L 603 4863 \n","z\n","\" transform=\"scale(0.015625)\"/>\n","       <path id=\"DejaVuSans-73\" d=\"M 2834 3397 \n","L 2834 2853 \n","Q 2591 2978 2328 3040 \n","Q 2066 3103 1784 3103 \n","Q 1356 3103 1142 2972 \n","Q 928 2841 928 2578 \n","Q 928 2378 1081 2264 \n","Q 1234 2150 1697 2047 \n","L 1894 2003 \n","Q 2506 1872 2764 1633 \n","Q 3022 1394 3022 966 \n","Q 3022 478 2636 193 \n","Q 2250 -91 1575 -91 \n","Q 1294 -91 989 -36 \n","Q 684 19 347 128 \n","L 347 722 \n","Q 666 556 975 473 \n","Q 1284 391 1588 391 \n","Q 1994 391 2212 530 \n","Q 2431 669 2431 922 \n","Q 2431 1156 2273 1281 \n","Q 2116 1406 1581 1522 \n","L 1381 1569 \n","Q 847 1681 609 1914 \n","Q 372 2147 372 2553 \n","Q 372 3047 722 3315 \n","Q 1072 3584 1716 3584 \n","Q 2034 3584 2315 3537 \n","Q 2597 3491 2834 3397 \n","z\n","\" transform=\"scale(0.015625)\"/>\n","      </defs>\n","      <use xlink:href=\"#DejaVuSans-74\"/>\n","      <use xlink:href=\"#DejaVuSans-72\" x=\"39.208984\"/>\n","      <use xlink:href=\"#DejaVuSans-61\" x=\"80.322266\"/>\n","      <use xlink:href=\"#DejaVuSans-69\" x=\"141.601562\"/>\n","      <use xlink:href=\"#DejaVuSans-6e\" x=\"169.384766\"/>\n","      <use xlink:href=\"#DejaVuSans-20\" x=\"232.763672\"/>\n","      <use xlink:href=\"#DejaVuSans-6c\" x=\"264.550781\"/>\n","      <use xlink:href=\"#DejaVuSans-6f\" x=\"292.333984\"/>\n","      <use xlink:href=\"#DejaVuSans-73\" x=\"353.515625\"/>\n","      <use xlink:href=\"#DejaVuSans-73\" x=\"405.615234\"/>\n","     </g>\n","    </g>\n","    <g id=\"line2d_21\">\n","     <path d=\"M 142.634375 74.409375 \n","L 152.634375 74.409375 \n","L 162.634375 74.409375 \n","\" style=\"fill: none; stroke-dasharray: 5.55,2.4; stroke-dashoffset: 0; stroke: #bf00bf; stroke-width: 1.5\"/>\n","    </g>\n","    <g id=\"text_11\">\n","     <!-- train acc -->\n","     <g transform=\"translate(170.634375 77.909375)scale(0.1 -0.1)\">\n","      <use xlink:href=\"#DejaVuSans-74\"/>\n","      <use xlink:href=\"#DejaVuSans-72\" x=\"39.208984\"/>\n","      <use xlink:href=\"#DejaVuSans-61\" x=\"80.322266\"/>\n","      <use xlink:href=\"#DejaVuSans-69\" x=\"141.601562\"/>\n","      <use xlink:href=\"#DejaVuSans-6e\" x=\"169.384766\"/>\n","      <use xlink:href=\"#DejaVuSans-20\" x=\"232.763672\"/>\n","      <use xlink:href=\"#DejaVuSans-61\" x=\"264.550781\"/>\n","      <use xlink:href=\"#DejaVuSans-63\" x=\"325.830078\"/>\n","      <use xlink:href=\"#DejaVuSans-63\" x=\"380.810547\"/>\n","     </g>\n","    </g>\n","    <g id=\"line2d_22\">\n","     <path d=\"M 142.634375 89.0875 \n","L 152.634375 89.0875 \n","L 162.634375 89.0875 \n","\" style=\"fill: none; stroke-dasharray: 9.6,2.4,1.5,2.4; stroke-dashoffset: 0; stroke: #008000; stroke-width: 1.5\"/>\n","    </g>\n","    <g id=\"text_12\">\n","     <!-- test acc -->\n","     <g transform=\"translate(170.634375 92.5875)scale(0.1 -0.1)\">\n","      <use xlink:href=\"#DejaVuSans-74\"/>\n","      <use xlink:href=\"#DejaVuSans-65\" x=\"39.208984\"/>\n","      <use xlink:href=\"#DejaVuSans-73\" x=\"100.732422\"/>\n","      <use xlink:href=\"#DejaVuSans-74\" x=\"152.832031\"/>\n","      <use xlink:href=\"#DejaVuSans-20\" x=\"192.041016\"/>\n","      <use xlink:href=\"#DejaVuSans-61\" x=\"223.828125\"/>\n","      <use xlink:href=\"#DejaVuSans-63\" x=\"285.107422\"/>\n","      <use xlink:href=\"#DejaVuSans-63\" x=\"340.087891\"/>\n","     </g>\n","    </g>\n","   </g>\n","  </g>\n"," </g>\n"," <defs>\n","  <clipPath id=\"p1f1a59b8d9\">\n","   <rect x=\"30.103125\" y=\"7.2\" width=\"195.3\" height=\"135.9\"/>\n","  </clipPath>\n"," </defs>\n","</svg>\n"],"text/plain":["<Figure size 252x180 with 1 Axes>"]},"metadata":{"needs_background":"light"},"output_type":"display_data"}],"source":["train_iter, test_iter = d2l.load_data_fashion_mnist(batch_size)\n","d2l.train_ch3(net, train_iter, test_iter, loss, num_epochs, trainer)"],"id":"UYsrUQA20i98"},{"cell_type":"markdown","metadata":{"origin_pos":0,"id":"0d786ba9"},"source":["# 模型选择、欠拟合和过拟合\n","\n","作为机器学习科学家，我们的目标是发现*模式*（pattern）。\n","但是，我们如何才能确定模型是真正发现了一种泛化的模式，\n","而不是简单地记住了数据呢？\n","例如，我们想要在患者的基因数据与痴呆状态之间寻找模式，\n","其中标签是从集合$\\{\\text{痴呆}, \\text{轻度认知障碍}, \\text{健康}\\}$中提取的。\n","因为基因可以唯一确定每个个体（不考虑双胞胎），\n","所以在这个任务中是有可能记住整个数据集的。\n","\n","我们不想让模型只会做这样的事情：“那是鲍勃！我记得他！他有痴呆症！”。\n","原因很简单：当我们将来部署该模型时，模型需要判断从未见过的患者。\n","只有当模型真正发现了一种泛化模式时，才会作出有效的预测。\n","\n","更正式地说，我们的目标是发现某些模式，\n","这些模式捕捉到了我们训练集潜在总体的规律。\n","如果成功做到了这点，即使是对以前从未遇到过的个体，\n","模型也可以成功地评估风险。\n","如何发现可以泛化的模式是机器学习的根本问题。\n","\n","困难在于，当我们训练模型时，我们只能访问数据中的小部分样本。\n","最大的公开图像数据集包含大约一百万张图像。\n","而在大部分时候，我们只能从数千或数万个数据样本中学习。\n","在大型医院系统中，我们可能会访问数十万份医疗记录。\n","当我们使用有限的样本时，可能会遇到这样的问题：\n","当收集到更多的数据时，会发现之前找到的明显关系并不成立。\n","\n","将模型在训练数据上拟合的比在潜在分布中更接近的现象称为*过拟合*（overfitting），\n","用于对抗过拟合的技术称为*正则化*（regularization）。\n","在前面的章节中，你可能在用Fashion-MNIST数据集做实验时已经观察到了这种过拟合现象。\n","在实验中调整模型架构或超参数时，你会发现：\n","如果有足够多的神经元、层数和训练迭代周期，\n","模型最终可以在训练集上达到完美的精度，此时测试集的准确性却下降了。\n","\n","## 训练误差和泛化误差\n","\n","为了进一步讨论这一现象，我们需要了解训练误差和泛化误差。\n","*训练误差*（training error）是指，\n","模型在训练数据集上计算得到的误差。\n","*泛化误差*（generalization error）是指，\n","模型应用在同样从原始样本的分布中抽取的无限多数据样本时，模型误差的期望。\n","\n","问题是，我们永远不能准确地计算出泛化误差。\n","这是因为无限多的数据样本是一个虚构的对象。\n","在实际中，我们只能通过将模型应用于一个独立的测试集来估计泛化误差，\n","该测试集由随机选取的、未曾在训练集中出现的数据样本构成。\n","\n","下面的三个思维实验将有助于更好地说明这种情况。\n","假设一个大学生正在努力准备期末考试。\n","一个勤奋的学生会努力做好练习，并利用往年的考试题目来测试自己的能力。\n","尽管如此，在过去的考试题目上取得好成绩并不能保证他会在真正考试时发挥出色。\n","例如，学生可能试图通过死记硬背考题的答案来做准备。\n","他甚至可以完全记住过去考试的答案。\n","另一名学生可能会通过试图理解给出某些答案的原因来做准备。\n","在大多数情况下，后者会考得更好。\n","\n","类似地，考虑一个简单地使用查表法来回答问题的模型。\n","如果允许的输入集合是离散的并且相当小，\n","那么也许在查看许多训练样本后，该方法将执行得很好。\n","但当这个模型面对从未见过的例子时，它表现的可能比随机猜测好不到哪去。\n","这是因为输入空间太大了，远远不可能记住每一个可能的输入所对应的答案。\n","例如，考虑$28\\times28$的灰度图像。\n","如果每个像素可以取$256$个灰度值中的一个，\n","则有$256^{784}$个可能的图像。\n","这意味着指甲大小的低分辨率灰度图像的数量比宇宙中的原子要多得多。\n","即使我们可能遇到这样的数据，我们也不可能存储整个查找表。\n","\n","最后，考虑对掷硬币的结果（类别0：正面，类别1：反面）进行分类的问题。\n","假设硬币是公平的，无论我们想出什么算法，泛化误差始终是$\\frac{1}{2}$。\n","然而，对于大多数算法，我们应该期望训练误差会更低（取决于运气）。\n","考虑数据集{0，1，1，1，0，1}。\n","我们的算法不需要额外的特征，将倾向于总是预测*多数类*，\n","从我们有限的样本来看，它似乎是1占主流。\n","在这种情况下，总是预测类1的模型将产生$\\frac{1}{3}$的误差，\n","这比我们的泛化误差要好得多。\n","当我们逐渐增加数据量，正面比例明显偏离$\\frac{1}{2}$的可能性将会降低，\n","我们的训练误差将与泛化误差相匹配。\n","\n","### 统计学习理论\n","\n","由于泛化是机器学习中的基本问题，\n","许多数学家和理论家毕生致力于研究描述这一现象的形式理论。\n","在[同名定理（eponymous theorem）](https://en.wikipedia.org/wiki/Glivenko%E2%80%93Cantelli_theorem)中，\n","格里文科和坎特利推导出了训练误差收敛到泛化误差的速率。\n","在一系列开创性的论文中，\n","[Vapnik和Chervonenkis](https://en.wikipedia.org/wiki/Vapnik%E2%80%93Chervonenkis_theory)\n","将这一理论扩展到更一般种类的函数。\n","这项工作为统计学习理论奠定了基础。\n","\n","在我们目前已探讨、并将在之后继续探讨的监督学习情景中，\n","我们假设训练数据和测试数据都是从相同的分布中独立提取的。\n","这通常被称为*独立同分布假设*（i.i.d. assumption），\n","这意味着对数据进行采样的过程没有进行“记忆”。\n","换句话说，抽取的第2个样本和第3个样本的相关性，\n","并不比抽取的第2个样本和第200万个样本的相关性更强。\n","\n","要成为一名优秀的机器学习科学家需要具备批判性思考能力。\n","你应该已经从这个假设中找出漏洞，即很容易找出假设失效的情况。\n","如果我们根据从加州大学旧金山分校医学中心的患者数据训练死亡风险预测模型，\n","并将其应用于马萨诸塞州综合医院的患者数据，结果会怎么样？\n","这两个数据的分布可能不完全一样。\n","此外，抽样过程可能与时间有关。\n","比如当我们对微博的主题进行分类时，\n","新闻周期会使得正在讨论的话题产生时间依赖性，从而违反独立性假设。\n","\n","有时候我们即使轻微违背独立同分布假设，模型仍将继续运行得非常好。\n","比如，我们有许多有用的工具已经应用于现实，如人脸识别、语音识别和语言翻译。\n","毕竟，几乎所有现实的应用都至少涉及到一些违背独立同分布假设的情况。\n","\n","有些违背独立同分布假设的行为肯定会带来麻烦。\n","比如，我们试图只用来自大学生的人脸数据来训练一个人脸识别系统，\n","然后想要用它来监测疗养院中的老人。\n","这不太可能有效，因为大学生看起来往往与老年人有很大的不同。\n","\n","在接下来的章节中，我们将讨论因违背独立同分布假设而引起的问题。\n","目前，即使认为独立同分布假设是理所当然的，理解泛化性也是一个困难的问题。\n","此外，能够解释深层神经网络泛化性能的理论基础，\n","也仍在继续困扰着学习理论领域最伟大的学者们。\n","\n","当我们训练模型时，我们试图找到一个能够尽可能拟合训练数据的函数。\n","但是如果它执行地“太好了”，而不能对看不见的数据做到很好泛化，就会导致过拟合。\n","这种情况正是我们想要避免或控制的。\n","深度学习中有许多启发式的技术旨在防止过拟合。\n","\n","### 模型复杂性\n","\n","当我们有简单的模型和大量的数据时，我们期望泛化误差与训练误差相近。\n","当我们有更复杂的模型和更少的样本时，我们预计训练误差会下降，但泛化误差会增大。\n","模型复杂性由什么构成是一个复杂的问题。\n","一个模型是否能很好地泛化取决于很多因素。\n","例如，具有更多参数的模型可能被认为更复杂，\n","参数有更大取值范围的模型可能更为复杂。\n","通常对于神经网络，我们认为需要更多训练迭代的模型比较复杂，\n","而需要“早停”（early stopping）的模型（即较少训练迭代周期）就不那么复杂。\n","\n","我们很难比较本质上不同大类的模型之间（例如，决策树与神经网络）的复杂性。\n","就目前而言，一条简单的经验法则相当有用：\n","统计学家认为，能够轻松解释任意事实的模型是复杂的，\n","而表达能力有限但仍能很好地解释数据的模型可能更有现实用途。\n","在哲学上，这与波普尔的科学理论的可证伪性标准密切相关：\n","如果一个理论能拟合数据，且有具体的测试可以用来证明它是错误的，那么它就是好的。\n","这一点很重要，因为所有的统计估计都是*事后归纳*。\n","也就是说，我们在观察事实之后进行估计，因此容易受到相关谬误的影响。\n","目前，我们将把哲学放在一边，坚持更切实的问题。\n","\n","在本节中，为了给你一些直观的印象，我们将重点介绍几个倾向于影响模型泛化的因素：\n","\n","1. 可调整参数的数量。当可调整参数的数量（有时称为*自由度*）很大时，模型往往更容易过拟合。\n","1. 参数采用的值。当权重的取值范围较大时，模型可能更容易过拟合。\n","1. 训练样本的数量。即使你的模型很简单，也很容易过拟合只包含一两个样本的数据集。而过拟合一个有数百万个样本的数据集则需要一个极其灵活的模型。\n","\n","## 模型选择\n","\n","在机器学习中，我们通常在评估几个候选模型后选择最终的模型。\n","这个过程叫做*模型选择*。\n","有时，需要进行比较的模型在本质上是完全不同的（比如，决策树与线性模型）。\n","又有时，我们需要比较不同的超参数设置下的同一类模型。\n","\n","例如，训练多层感知机模型时，我们可能希望比较具有\n","不同数量的隐藏层、不同数量的隐藏单元以及不同的激活函数组合的模型。\n","为了确定候选模型中的最佳模型，我们通常会使用验证集。\n","\n","### 验证集\n","\n","原则上，在我们确定所有的超参数之前，我们不希望用到测试集。\n","如果我们在模型选择过程中使用测试数据，可能会有过拟合测试数据的风险，那就麻烦大了。\n","如果我们过拟合了训练数据，还可以在测试数据上的评估来判断过拟合。\n","但是如果我们过拟合了测试数据，我们又该怎么知道呢？\n","\n","因此，我们决不能依靠测试数据进行模型选择。\n","然而，我们也不能仅仅依靠训练数据来选择模型，因为我们无法估计训练数据的泛化误差。\n","\n","在实际应用中，情况变得更加复杂。\n","虽然理想情况下我们只会使用测试数据一次，\n","以评估最好的模型或比较一些模型效果，但现实是测试数据很少在使用一次后被丢弃。\n","我们很少能有充足的数据来对每一轮实验采用全新测试集。\n","\n","解决此问题的常见做法是将我们的数据分成三份，\n","除了训练和测试数据集之外，还增加一个*验证数据集*（validation dataset），\n","也叫*验证集*（validation set）。\n","但现实是验证数据和测试数据之间的边界模糊得令人担忧。\n","除非另有明确说明，否则在这本书的实验中，\n","我们实际上是在使用应该被正确地称为训练数据和验证数据的数据集，\n","并没有真正的测试数据集。\n","因此，书中每次实验报告的准确度都是验证集准确度，而不是测试集准确度。\n","\n","### $K$折交叉验证\n","\n","当训练数据稀缺时，我们甚至可能无法提供足够的数据来构成一个合适的验证集。\n","这个问题的一个流行的解决方案是采用$K$*折交叉验证*。\n","这里，原始训练数据被分成$K$个不重叠的子集。\n","然后执行$K$次模型训练和验证，每次在$K-1$个子集上进行训练，\n","并在剩余的一个子集（在该轮中没有用于训练的子集）上进行验证。\n","最后，通过对$K$次实验的结果取平均来估计训练和验证误差。\n","\n","## 欠拟合还是过拟合？\n","\n","当我们比较训练和验证误差时，我们要注意两种常见的情况。\n","首先，我们要注意这样的情况：训练误差和验证误差都很严重，\n","但它们之间仅有一点差距。\n","如果模型不能降低训练误差，这可能意味着模型过于简单（即表达能力不足），\n","无法捕获试图学习的模式。\n","此外，由于我们的训练和验证误差之间的*泛化误差*很小，\n","我们有理由相信可以用一个更复杂的模型降低训练误差。\n","这种现象被称为*欠拟合*（underfitting）。\n","\n","另一方面，当我们的训练误差明显低于验证误差时要小心，\n","这表明严重的*过拟合*（overfitting）。\n","注意，*过拟合*并不总是一件坏事。\n","特别是在深度学习领域，众所周知，\n","最好的预测模型在训练数据上的表现往往比在保留（验证）数据上好得多。\n","最终，我们通常更关心验证误差，而不是训练误差和验证误差之间的差距。\n","\n","是否过拟合或欠拟合可能取决于模型复杂性和可用训练数据集的大小，\n","这两个点将在下面进行讨论。\n","\n","### 模型复杂性\n","\n","为了说明一些关于过拟合和模型复杂性的经典直觉，\n","我们给出一个多项式的例子。\n","给定由单个特征$x$和对应实数标签$y$组成的训练数据，\n","我们试图找到下面的$d$阶多项式来估计标签$y$。\n","\n","$$\\hat{y}= \\sum_{i=0}^d x^i w_i$$\n","\n","这只是一个线性回归问题，我们的特征是$x$的幂给出的，\n","模型的权重是$w_i$给出的，偏置是$w_0$给出的\n","（因为对于所有的$x$都有$x^0 = 1$）。\n","由于这只是一个线性回归问题，我们可以使用平方误差作为我们的损失函数。\n","\n","高阶多项式函数比低阶多项式函数复杂得多。\n","高阶多项式的参数较多，模型函数的选择范围较广。\n","因此在固定训练数据集的情况下，\n","高阶多项式函数相对于低阶多项式的训练误差应该始终更低（最坏也是相等）。\n","事实上，当数据样本包含了$x$的不同值时，\n","函数阶数等于数据样本数量的多项式函数可以完美拟合训练集。\n","在 :numref:`fig_capacity_vs_error`中，\n","我们直观地描述了多项式的阶数和欠拟合与过拟合之间的关系。\n","\n","\n","![模型复杂度对欠拟合和过拟合的影响](../img/capacity-vs-error.svg)\n",":label:`fig_capacity_vs_error`\n","\n","### 数据集大小\n","\n","另一个重要因素是数据集的大小。\n","训练数据集中的样本越少，我们就越有可能（且更严重地）过拟合。\n","随着训练数据量的增加，泛化误差通常会减小。\n","此外，一般来说，更多的数据不会有什么坏处。\n","对于固定的任务和数据分布，模型复杂性和数据集大小之间通常存在关系。\n","给出更多的数据，我们可能会尝试拟合一个更复杂的模型。\n","能够拟合更复杂的模型可能是有益的。\n","如果没有足够的数据，简单的模型可能更有用。\n","对于许多任务，深度学习只有在有数千个训练样本时才优于线性模型。\n","从一定程度上来说，深度学习目前的生机要归功于\n","廉价存储、互联设备以及数字化经济带来的海量数据集。\n","\n","## 多项式回归\n","\n","我们现在可以(**通过多项式拟合来探索这些概念**)。\n"],"id":"0d786ba9"},{"cell_type":"code","execution_count":null,"metadata":{"execution":{"iopub.execute_input":"2022-07-31T02:20:39.059180Z","iopub.status.busy":"2022-07-31T02:20:39.058521Z","iopub.status.idle":"2022-07-31T02:20:40.964805Z","shell.execute_reply":"2022-07-31T02:20:40.963805Z"},"origin_pos":2,"tab":["pytorch"],"id":"d9436818"},"outputs":[],"source":["import math\n","import numpy as np\n","import torch\n","from torch import nn\n","from d2l import torch as d2l"],"id":"d9436818"},{"cell_type":"markdown","metadata":{"origin_pos":4,"id":"2c81b105"},"source":["### 生成数据集\n","\n","给定$x$，我们将[**使用以下三阶多项式来生成训练和测试数据的标签：**]\n","\n","(**$$y = 5 + 1.2x - 3.4\\frac{x^2}{2!} + 5.6 \\frac{x^3}{3!} + \\epsilon \\text{ where }\n","\\epsilon \\sim \\mathcal{N}(0, 0.1^2).$$**)\n","\n","噪声项$\\epsilon$服从均值为0且标准差为0.1的正态分布。\n","在优化的过程中，我们通常希望避免非常大的梯度值或损失值。\n","这就是我们将特征从$x^i$调整为$\\frac{x^i}{i!}$的原因，\n","这样可以避免很大的$i$带来的特别大的指数值。\n","我们将为训练集和测试集各生成100个样本。\n"],"id":"2c81b105"},{"cell_type":"code","execution_count":null,"metadata":{"execution":{"iopub.execute_input":"2022-07-31T02:20:40.968819Z","iopub.status.busy":"2022-07-31T02:20:40.968391Z","iopub.status.idle":"2022-07-31T02:20:40.976184Z","shell.execute_reply":"2022-07-31T02:20:40.975516Z"},"origin_pos":5,"tab":["pytorch"],"id":"9bd8645c"},"outputs":[],"source":["max_degree = 20  # 多项式的最大阶数\n","n_train, n_test = 100, 100  # 训练和测试数据集大小\n","true_w = np.zeros(max_degree)  # 分配大量的空间\n","true_w[0:4] = np.array([5, 1.2, -3.4, 5.6])\n","\n","features = np.random.normal(size=(n_train + n_test, 1))\n","np.random.shuffle(features)\n","poly_features = np.power(features, np.arange(max_degree).reshape(1, -1))\n","for i in range(max_degree):\n","    poly_features[:, i] /= math.gamma(i + 1)  # gamma(n)=(n-1)!\n","# labels的维度:(n_train+n_test,)\n","labels = np.dot(poly_features, true_w)\n","labels += np.random.normal(scale=0.1, size=labels.shape)"],"id":"9bd8645c"},{"cell_type":"markdown","metadata":{"origin_pos":6,"id":"d8229ac5"},"source":["同样，存储在`poly_features`中的单项式由gamma函数重新缩放，\n","其中$\\Gamma(n)=(n-1)!$。\n","从生成的数据集中[**查看一下前2个样本**]，\n","第一个值是与偏置相对应的常量特征。\n"],"id":"d8229ac5"},{"cell_type":"code","execution_count":null,"metadata":{"execution":{"iopub.execute_input":"2022-07-31T02:20:40.979295Z","iopub.status.busy":"2022-07-31T02:20:40.978874Z","iopub.status.idle":"2022-07-31T02:20:40.982884Z","shell.execute_reply":"2022-07-31T02:20:40.982264Z"},"origin_pos":7,"tab":["pytorch"],"id":"5cbadcea"},"outputs":[],"source":["# NumPy ndarray转换为tensor\n","true_w, features, poly_features, labels = [torch.tensor(x, dtype=\n","    torch.float32) for x in [true_w, features, poly_features, labels]]"],"id":"5cbadcea"},{"cell_type":"code","execution_count":null,"metadata":{"execution":{"iopub.execute_input":"2022-07-31T02:20:40.985840Z","iopub.status.busy":"2022-07-31T02:20:40.985471Z","iopub.status.idle":"2022-07-31T02:20:40.994230Z","shell.execute_reply":"2022-07-31T02:20:40.993320Z"},"origin_pos":8,"tab":["pytorch"],"id":"0d99c6ae","outputId":"84e39440-f8d8-4f35-f54c-d958b51e3409"},"outputs":[{"data":{"text/plain":["(tensor([[0.8069],\n","         [0.1614]]),\n"," tensor([[1.0000e+00, 8.0685e-01, 3.2550e-01, 8.7544e-02, 1.7659e-02, 2.8496e-03,\n","          3.8320e-04, 4.4169e-05, 4.4548e-06, 3.9937e-07, 3.2223e-08, 2.3636e-09,\n","          1.5892e-10, 9.8635e-12, 5.6845e-13, 3.0577e-14, 1.5419e-15, 7.3184e-17,\n","          3.2805e-18, 1.3931e-19],\n","         [1.0000e+00, 1.6137e-01, 1.3020e-02, 7.0031e-04, 2.8252e-05, 9.1177e-07,\n","          2.4522e-08, 5.6528e-10, 1.1402e-11, 2.0444e-13, 3.2989e-15, 4.8394e-17,\n","          6.5077e-19, 8.0778e-21, 9.3107e-23, 1.0016e-24, 1.0102e-26, 9.5887e-29,\n","          8.5961e-31, 7.3007e-33]]),\n"," tensor([5.2975, 5.0520]))"]},"execution_count":4,"metadata":{},"output_type":"execute_result"}],"source":["features[:2], poly_features[:2, :], labels[:2]"],"id":"0d99c6ae"},{"cell_type":"markdown","metadata":{"origin_pos":9,"id":"6bd9a46d"},"source":["### 对模型进行训练和测试\n","\n","首先让我们[**实现一个函数来评估模型在给定数据集上的损失**]。\n"],"id":"6bd9a46d"},{"cell_type":"code","execution_count":null,"metadata":{"execution":{"iopub.execute_input":"2022-07-31T02:20:40.997484Z","iopub.status.busy":"2022-07-31T02:20:40.997079Z","iopub.status.idle":"2022-07-31T02:20:41.002789Z","shell.execute_reply":"2022-07-31T02:20:41.001922Z"},"origin_pos":11,"tab":["pytorch"],"id":"74393954"},"outputs":[],"source":["def evaluate_loss(net, data_iter, loss):  \n","    \"\"\"评估给定数据集上模型的损失\"\"\"\n","    metric = d2l.Accumulator(2)  # 损失的总和,样本数量\n","    for X, y in data_iter:\n","        out = net(X)\n","        y = y.reshape(out.shape)\n","        l = loss(out, y)\n","        metric.add(l.sum(), l.numel())\n","    return metric[0] / metric[1]"],"id":"74393954"},{"cell_type":"markdown","metadata":{"origin_pos":12,"id":"b45b7fa0"},"source":["现在[**定义训练函数**]。\n"],"id":"b45b7fa0"},{"cell_type":"code","execution_count":null,"metadata":{"execution":{"iopub.execute_input":"2022-07-31T02:20:41.005969Z","iopub.status.busy":"2022-07-31T02:20:41.005429Z","iopub.status.idle":"2022-07-31T02:20:41.014008Z","shell.execute_reply":"2022-07-31T02:20:41.013098Z"},"origin_pos":14,"tab":["pytorch"],"id":"12e3115d"},"outputs":[],"source":["def train(train_features, test_features, train_labels, test_labels,\n","          num_epochs=400):\n","    loss = nn.MSELoss(reduction='none')\n","    input_shape = train_features.shape[-1]\n","    # 不设置偏置，因为我们已经在多项式中实现了它\n","    net = nn.Sequential(nn.Linear(input_shape, 1, bias=False))\n","    batch_size = min(10, train_labels.shape[0])\n","    train_iter = d2l.load_array((train_features, train_labels.reshape(-1,1)),\n","                                batch_size)\n","    test_iter = d2l.load_array((test_features, test_labels.reshape(-1,1)),\n","                               batch_size, is_train=False)\n","    trainer = torch.optim.SGD(net.parameters(), lr=0.01)\n","    animator = d2l.Animator(xlabel='epoch', ylabel='loss', yscale='log',\n","                            xlim=[1, num_epochs], ylim=[1e-3, 1e2],\n","                            legend=['train', 'test'])\n","    for epoch in range(num_epochs):\n","        d2l.train_epoch_ch3(net, train_iter, loss, trainer)\n","        if epoch == 0 or (epoch + 1) % 20 == 0:\n","            animator.add(epoch + 1, (evaluate_loss(net, train_iter, loss),\n","                                     evaluate_loss(net, test_iter, loss)))\n","    print('weight:', net[0].weight.data.numpy())"],"id":"12e3115d"},{"cell_type":"markdown","metadata":{"origin_pos":16,"id":"5e459c61"},"source":["### [**三阶多项式函数拟合(正常)**]\n","\n","我们将首先使用三阶多项式函数，它与数据生成函数的阶数相同。\n","结果表明，该模型能有效降低训练损失和测试损失。\n","学习到的模型参数也接近真实值$w = [5, 1.2, -3.4, 5.6]$。\n"],"id":"5e459c61"},{"cell_type":"code","execution_count":null,"metadata":{"execution":{"iopub.execute_input":"2022-07-31T02:20:41.017521Z","iopub.status.busy":"2022-07-31T02:20:41.016837Z","iopub.status.idle":"2022-07-31T02:20:53.408931Z","shell.execute_reply":"2022-07-31T02:20:53.408267Z"},"origin_pos":17,"tab":["pytorch"],"id":"4120f7d7","outputId":"7daf2eae-a2f7-4110-fab3-50e9e09e1250"},"outputs":[{"name":"stdout","output_type":"stream","text":["weight: [[ 5.0007887  1.188556  -3.4069114  5.598693 ]]\n"]},{"data":{"image/svg+xml":["<?xml version=\"1.0\" encoding=\"utf-8\" standalone=\"no\"?>\n","<!DOCTYPE svg PUBLIC \"-//W3C//DTD SVG 1.1//EN\"\n","  \"http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd\">\n","<svg xmlns:xlink=\"http://www.w3.org/1999/xlink\" width=\"263.421875pt\" height=\"184.455469pt\" viewBox=\"0 0 263.421875 184.455469\" xmlns=\"http://www.w3.org/2000/svg\" version=\"1.1\">\n"," <metadata>\n","  <rdf:RDF xmlns:dc=\"http://purl.org/dc/elements/1.1/\" xmlns:cc=\"http://creativecommons.org/ns#\" xmlns:rdf=\"http://www.w3.org/1999/02/22-rdf-syntax-ns#\">\n","   <cc:Work>\n","    <dc:type rdf:resource=\"http://purl.org/dc/dcmitype/StillImage\"/>\n","    <dc:date>2022-07-31T02:20:53.277337</dc:date>\n","    <dc:format>image/svg+xml</dc:format>\n","    <dc:creator>\n","     <cc:Agent>\n","      <dc:title>Matplotlib v3.5.1, https://matplotlib.org/</dc:title>\n","     </cc:Agent>\n","    </dc:creator>\n","   </cc:Work>\n","  </rdf:RDF>\n"," </metadata>\n"," <defs>\n","  <style type=\"text/css\">*{stroke-linejoin: round; stroke-linecap: butt}</style>\n"," </defs>\n"," <g id=\"figure_1\">\n","  <g id=\"patch_1\">\n","   <path d=\"M 0 184.455469 \n","L 263.421875 184.455469 \n","L 263.421875 0 \n","L 0 0 \n","L 0 184.455469 \n","z\n","\" style=\"fill: none\"/>\n","  </g>\n","  <g id=\"axes_1\">\n","   <g id=\"patch_2\">\n","    <path d=\"M 51.378125 146.899219 \n","L 246.678125 146.899219 \n","L 246.678125 10.999219 \n","L 51.378125 10.999219 \n","z\n","\" style=\"fill: #ffffff\"/>\n","   </g>\n","   <g id=\"matplotlib.axis_1\">\n","    <g id=\"xtick_1\">\n","     <g id=\"line2d_1\">\n","      <path d=\"M 99.83602 146.899219 \n","L 99.83602 10.999219 \n","\" clip-path=\"url(#pa12fe637e7)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n","     </g>\n","     <g id=\"line2d_2\">\n","      <defs>\n","       <path id=\"me7b99da713\" d=\"M 0 0 \n","L 0 3.5 \n","\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n","      </defs>\n","      <g>\n","       <use xlink:href=\"#me7b99da713\" x=\"99.83602\" y=\"146.899219\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n","      </g>\n","     </g>\n","     <g id=\"text_1\">\n","      <!-- 100 -->\n","      <g transform=\"translate(90.29227 161.497656)scale(0.1 -0.1)\">\n","       <defs>\n","        <path id=\"DejaVuSans-31\" d=\"M 794 531 \n","L 1825 531 \n","L 1825 4091 \n","L 703 3866 \n","L 703 4441 \n","L 1819 4666 \n","L 2450 4666 \n","L 2450 531 \n","L 3481 531 \n","L 3481 0 \n","L 794 0 \n","L 794 531 \n","z\n","\" transform=\"scale(0.015625)\"/>\n","        <path id=\"DejaVuSans-30\" d=\"M 2034 4250 \n","Q 1547 4250 1301 3770 \n","Q 1056 3291 1056 2328 \n","Q 1056 1369 1301 889 \n","Q 1547 409 2034 409 \n","Q 2525 409 2770 889 \n","Q 3016 1369 3016 2328 \n","Q 3016 3291 2770 3770 \n","Q 2525 4250 2034 4250 \n","z\n","M 2034 4750 \n","Q 2819 4750 3233 4129 \n","Q 3647 3509 3647 2328 \n","Q 3647 1150 3233 529 \n","Q 2819 -91 2034 -91 \n","Q 1250 -91 836 529 \n","Q 422 1150 422 2328 \n","Q 422 3509 836 4129 \n","Q 1250 4750 2034 4750 \n","z\n","\" transform=\"scale(0.015625)\"/>\n","       </defs>\n","       <use xlink:href=\"#DejaVuSans-31\"/>\n","       <use xlink:href=\"#DejaVuSans-30\" x=\"63.623047\"/>\n","       <use xlink:href=\"#DejaVuSans-30\" x=\"127.246094\"/>\n","      </g>\n","     </g>\n","    </g>\n","    <g id=\"xtick_2\">\n","     <g id=\"line2d_3\">\n","      <path d=\"M 148.783388 146.899219 \n","L 148.783388 10.999219 \n","\" clip-path=\"url(#pa12fe637e7)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n","     </g>\n","     <g id=\"line2d_4\">\n","      <g>\n","       <use xlink:href=\"#me7b99da713\" x=\"148.783388\" y=\"146.899219\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n","      </g>\n","     </g>\n","     <g id=\"text_2\">\n","      <!-- 200 -->\n","      <g transform=\"translate(139.239638 161.497656)scale(0.1 -0.1)\">\n","       <defs>\n","        <path id=\"DejaVuSans-32\" d=\"M 1228 531 \n","L 3431 531 \n","L 3431 0 \n","L 469 0 \n","L 469 531 \n","Q 828 903 1448 1529 \n","Q 2069 2156 2228 2338 \n","Q 2531 2678 2651 2914 \n","Q 2772 3150 2772 3378 \n","Q 2772 3750 2511 3984 \n","Q 2250 4219 1831 4219 \n","Q 1534 4219 1204 4116 \n","Q 875 4013 500 3803 \n","L 500 4441 \n","Q 881 4594 1212 4672 \n","Q 1544 4750 1819 4750 \n","Q 2544 4750 2975 4387 \n","Q 3406 4025 3406 3419 \n","Q 3406 3131 3298 2873 \n","Q 3191 2616 2906 2266 \n","Q 2828 2175 2409 1742 \n","Q 1991 1309 1228 531 \n","z\n","\" transform=\"scale(0.015625)\"/>\n","       </defs>\n","       <use xlink:href=\"#DejaVuSans-32\"/>\n","       <use xlink:href=\"#DejaVuSans-30\" x=\"63.623047\"/>\n","       <use xlink:href=\"#DejaVuSans-30\" x=\"127.246094\"/>\n","      </g>\n","     </g>\n","    </g>\n","    <g id=\"xtick_3\">\n","     <g id=\"line2d_5\">\n","      <path d=\"M 197.730757 146.899219 \n","L 197.730757 10.999219 \n","\" clip-path=\"url(#pa12fe637e7)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n","     </g>\n","     <g id=\"line2d_6\">\n","      <g>\n","       <use xlink:href=\"#me7b99da713\" x=\"197.730757\" y=\"146.899219\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n","      </g>\n","     </g>\n","     <g id=\"text_3\">\n","      <!-- 300 -->\n","      <g transform=\"translate(188.187007 161.497656)scale(0.1 -0.1)\">\n","       <defs>\n","        <path id=\"DejaVuSans-33\" d=\"M 2597 2516 \n","Q 3050 2419 3304 2112 \n","Q 3559 1806 3559 1356 \n","Q 3559 666 3084 287 \n","Q 2609 -91 1734 -91 \n","Q 1441 -91 1130 -33 \n","Q 819 25 488 141 \n","L 488 750 \n","Q 750 597 1062 519 \n","Q 1375 441 1716 441 \n","Q 2309 441 2620 675 \n","Q 2931 909 2931 1356 \n","Q 2931 1769 2642 2001 \n","Q 2353 2234 1838 2234 \n","L 1294 2234 \n","L 1294 2753 \n","L 1863 2753 \n","Q 2328 2753 2575 2939 \n","Q 2822 3125 2822 3475 \n","Q 2822 3834 2567 4026 \n","Q 2313 4219 1838 4219 \n","Q 1578 4219 1281 4162 \n","Q 984 4106 628 3988 \n","L 628 4550 \n","Q 988 4650 1302 4700 \n","Q 1616 4750 1894 4750 \n","Q 2613 4750 3031 4423 \n","Q 3450 4097 3450 3541 \n","Q 3450 3153 3228 2886 \n","Q 3006 2619 2597 2516 \n","z\n","\" transform=\"scale(0.015625)\"/>\n","       </defs>\n","       <use xlink:href=\"#DejaVuSans-33\"/>\n","       <use xlink:href=\"#DejaVuSans-30\" x=\"63.623047\"/>\n","       <use xlink:href=\"#DejaVuSans-30\" x=\"127.246094\"/>\n","      </g>\n","     </g>\n","    </g>\n","    <g id=\"xtick_4\">\n","     <g id=\"line2d_7\">\n","      <path d=\"M 246.678125 146.899219 \n","L 246.678125 10.999219 \n","\" clip-path=\"url(#pa12fe637e7)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n","     </g>\n","     <g id=\"line2d_8\">\n","      <g>\n","       <use xlink:href=\"#me7b99da713\" x=\"246.678125\" y=\"146.899219\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n","      </g>\n","     </g>\n","     <g id=\"text_4\">\n","      <!-- 400 -->\n","      <g transform=\"translate(237.134375 161.497656)scale(0.1 -0.1)\">\n","       <defs>\n","        <path id=\"DejaVuSans-34\" d=\"M 2419 4116 \n","L 825 1625 \n","L 2419 1625 \n","L 2419 4116 \n","z\n","M 2253 4666 \n","L 3047 4666 \n","L 3047 1625 \n","L 3713 1625 \n","L 3713 1100 \n","L 3047 1100 \n","L 3047 0 \n","L 2419 0 \n","L 2419 1100 \n","L 313 1100 \n","L 313 1709 \n","L 2253 4666 \n","z\n","\" transform=\"scale(0.015625)\"/>\n","       </defs>\n","       <use xlink:href=\"#DejaVuSans-34\"/>\n","       <use xlink:href=\"#DejaVuSans-30\" x=\"63.623047\"/>\n","       <use xlink:href=\"#DejaVuSans-30\" x=\"127.246094\"/>\n","      </g>\n","     </g>\n","    </g>\n","    <g id=\"text_5\">\n","     <!-- epoch -->\n","     <g transform=\"translate(133.8 175.175781)scale(0.1 -0.1)\">\n","      <defs>\n","       <path id=\"DejaVuSans-65\" d=\"M 3597 1894 \n","L 3597 1613 \n","L 953 1613 \n","Q 991 1019 1311 708 \n","Q 1631 397 2203 397 \n","Q 2534 397 2845 478 \n","Q 3156 559 3463 722 \n","L 3463 178 \n","Q 3153 47 2828 -22 \n","Q 2503 -91 2169 -91 \n","Q 1331 -91 842 396 \n","Q 353 884 353 1716 \n","Q 353 2575 817 3079 \n","Q 1281 3584 2069 3584 \n","Q 2775 3584 3186 3129 \n","Q 3597 2675 3597 1894 \n","z\n","M 3022 2063 \n","Q 3016 2534 2758 2815 \n","Q 2500 3097 2075 3097 \n","Q 1594 3097 1305 2825 \n","Q 1016 2553 972 2059 \n","L 3022 2063 \n","z\n","\" transform=\"scale(0.015625)\"/>\n","       <path id=\"DejaVuSans-70\" d=\"M 1159 525 \n","L 1159 -1331 \n","L 581 -1331 \n","L 581 3500 \n","L 1159 3500 \n","L 1159 2969 \n","Q 1341 3281 1617 3432 \n","Q 1894 3584 2278 3584 \n","Q 2916 3584 3314 3078 \n","Q 3713 2572 3713 1747 \n","Q 3713 922 3314 415 \n","Q 2916 -91 2278 -91 \n","Q 1894 -91 1617 61 \n","Q 1341 213 1159 525 \n","z\n","M 3116 1747 \n","Q 3116 2381 2855 2742 \n","Q 2594 3103 2138 3103 \n","Q 1681 3103 1420 2742 \n","Q 1159 2381 1159 1747 \n","Q 1159 1113 1420 752 \n","Q 1681 391 2138 391 \n","Q 2594 391 2855 752 \n","Q 3116 1113 3116 1747 \n","z\n","\" transform=\"scale(0.015625)\"/>\n","       <path id=\"DejaVuSans-6f\" d=\"M 1959 3097 \n","Q 1497 3097 1228 2736 \n","Q 959 2375 959 1747 \n","Q 959 1119 1226 758 \n","Q 1494 397 1959 397 \n","Q 2419 397 2687 759 \n","Q 2956 1122 2956 1747 \n","Q 2956 2369 2687 2733 \n","Q 2419 3097 1959 3097 \n","z\n","M 1959 3584 \n","Q 2709 3584 3137 3096 \n","Q 3566 2609 3566 1747 \n","Q 3566 888 3137 398 \n","Q 2709 -91 1959 -91 \n","Q 1206 -91 779 398 \n","Q 353 888 353 1747 \n","Q 353 2609 779 3096 \n","Q 1206 3584 1959 3584 \n","z\n","\" transform=\"scale(0.015625)\"/>\n","       <path id=\"DejaVuSans-63\" d=\"M 3122 3366 \n","L 3122 2828 \n","Q 2878 2963 2633 3030 \n","Q 2388 3097 2138 3097 \n","Q 1578 3097 1268 2742 \n","Q 959 2388 959 1747 \n","Q 959 1106 1268 751 \n","Q 1578 397 2138 397 \n","Q 2388 397 2633 464 \n","Q 2878 531 3122 666 \n","L 3122 134 \n","Q 2881 22 2623 -34 \n","Q 2366 -91 2075 -91 \n","Q 1284 -91 818 406 \n","Q 353 903 353 1747 \n","Q 353 2603 823 3093 \n","Q 1294 3584 2113 3584 \n","Q 2378 3584 2631 3529 \n","Q 2884 3475 3122 3366 \n","z\n","\" transform=\"scale(0.015625)\"/>\n","       <path id=\"DejaVuSans-68\" d=\"M 3513 2113 \n","L 3513 0 \n","L 2938 0 \n","L 2938 2094 \n","Q 2938 2591 2744 2837 \n","Q 2550 3084 2163 3084 \n","Q 1697 3084 1428 2787 \n","Q 1159 2491 1159 1978 \n","L 1159 0 \n","L 581 0 \n","L 581 4863 \n","L 1159 4863 \n","L 1159 2956 \n","Q 1366 3272 1645 3428 \n","Q 1925 3584 2291 3584 \n","Q 2894 3584 3203 3211 \n","Q 3513 2838 3513 2113 \n","z\n","\" transform=\"scale(0.015625)\"/>\n","      </defs>\n","      <use xlink:href=\"#DejaVuSans-65\"/>\n","      <use xlink:href=\"#DejaVuSans-70\" x=\"61.523438\"/>\n","      <use xlink:href=\"#DejaVuSans-6f\" x=\"125\"/>\n","      <use xlink:href=\"#DejaVuSans-63\" x=\"186.181641\"/>\n","      <use xlink:href=\"#DejaVuSans-68\" x=\"241.162109\"/>\n","     </g>\n","    </g>\n","   </g>\n","   <g id=\"matplotlib.axis_2\">\n","    <g id=\"ytick_1\">\n","     <g id=\"line2d_9\">\n","      <path d=\"M 51.378125 146.899219 \n","L 246.678125 146.899219 \n","\" clip-path=\"url(#pa12fe637e7)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n","     </g>\n","     <g id=\"line2d_10\">\n","      <defs>\n","       <path id=\"mcc687aa647\" d=\"M 0 0 \n","L -3.5 0 \n","\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n","      </defs>\n","      <g>\n","       <use xlink:href=\"#mcc687aa647\" x=\"51.378125\" y=\"146.899219\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n","      </g>\n","     </g>\n","     <g id=\"text_6\">\n","      <!-- $\\mathdefault{10^{-3}}$ -->\n","      <g transform=\"translate(20.878125 150.698438)scale(0.1 -0.1)\">\n","       <defs>\n","        <path id=\"DejaVuSans-2212\" d=\"M 678 2272 \n","L 4684 2272 \n","L 4684 1741 \n","L 678 1741 \n","L 678 2272 \n","z\n","\" transform=\"scale(0.015625)\"/>\n","       </defs>\n","       <use xlink:href=\"#DejaVuSans-31\" transform=\"translate(0 0.765625)\"/>\n","       <use xlink:href=\"#DejaVuSans-30\" transform=\"translate(63.623047 0.765625)\"/>\n","       <use xlink:href=\"#DejaVuSans-2212\" transform=\"translate(128.203125 39.046875)scale(0.7)\"/>\n","       <use xlink:href=\"#DejaVuSans-33\" transform=\"translate(186.855469 39.046875)scale(0.7)\"/>\n","      </g>\n","     </g>\n","    </g>\n","    <g id=\"ytick_2\">\n","     <g id=\"line2d_11\">\n","      <path d=\"M 51.378125 119.719219 \n","L 246.678125 119.719219 \n","\" clip-path=\"url(#pa12fe637e7)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n","     </g>\n","     <g id=\"line2d_12\">\n","      <g>\n","       <use xlink:href=\"#mcc687aa647\" x=\"51.378125\" y=\"119.719219\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n","      </g>\n","     </g>\n","     <g id=\"text_7\">\n","      <!-- $\\mathdefault{10^{-2}}$ -->\n","      <g transform=\"translate(20.878125 123.518438)scale(0.1 -0.1)\">\n","       <use xlink:href=\"#DejaVuSans-31\" transform=\"translate(0 0.765625)\"/>\n","       <use xlink:href=\"#DejaVuSans-30\" transform=\"translate(63.623047 0.765625)\"/>\n","       <use xlink:href=\"#DejaVuSans-2212\" transform=\"translate(128.203125 39.046875)scale(0.7)\"/>\n","       <use xlink:href=\"#DejaVuSans-32\" transform=\"translate(186.855469 39.046875)scale(0.7)\"/>\n","      </g>\n","     </g>\n","    </g>\n","    <g id=\"ytick_3\">\n","     <g id=\"line2d_13\">\n","      <path d=\"M 51.378125 92.539219 \n","L 246.678125 92.539219 \n","\" clip-path=\"url(#pa12fe637e7)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n","     </g>\n","     <g id=\"line2d_14\">\n","      <g>\n","       <use xlink:href=\"#mcc687aa647\" x=\"51.378125\" y=\"92.539219\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n","      </g>\n","     </g>\n","     <g id=\"text_8\">\n","      <!-- $\\mathdefault{10^{-1}}$ -->\n","      <g transform=\"translate(20.878125 96.338437)scale(0.1 -0.1)\">\n","       <use xlink:href=\"#DejaVuSans-31\" transform=\"translate(0 0.684375)\"/>\n","       <use xlink:href=\"#DejaVuSans-30\" transform=\"translate(63.623047 0.684375)\"/>\n","       <use xlink:href=\"#DejaVuSans-2212\" transform=\"translate(128.203125 38.965625)scale(0.7)\"/>\n","       <use xlink:href=\"#DejaVuSans-31\" transform=\"translate(186.855469 38.965625)scale(0.7)\"/>\n","      </g>\n","     </g>\n","    </g>\n","    <g id=\"ytick_4\">\n","     <g id=\"line2d_15\">\n","      <path d=\"M 51.378125 65.359219 \n","L 246.678125 65.359219 \n","\" clip-path=\"url(#pa12fe637e7)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n","     </g>\n","     <g id=\"line2d_16\">\n","      <g>\n","       <use xlink:href=\"#mcc687aa647\" x=\"51.378125\" y=\"65.359219\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n","      </g>\n","     </g>\n","     <g id=\"text_9\">\n","      <!-- $\\mathdefault{10^{0}}$ -->\n","      <g transform=\"translate(26.778125 69.158437)scale(0.1 -0.1)\">\n","       <use xlink:href=\"#DejaVuSans-31\" transform=\"translate(0 0.765625)\"/>\n","       <use xlink:href=\"#DejaVuSans-30\" transform=\"translate(63.623047 0.765625)\"/>\n","       <use xlink:href=\"#DejaVuSans-30\" transform=\"translate(128.203125 39.046875)scale(0.7)\"/>\n","      </g>\n","     </g>\n","    </g>\n","    <g id=\"ytick_5\">\n","     <g id=\"line2d_17\">\n","      <path d=\"M 51.378125 38.179219 \n","L 246.678125 38.179219 \n","\" clip-path=\"url(#pa12fe637e7)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n","     </g>\n","     <g id=\"line2d_18\">\n","      <g>\n","       <use xlink:href=\"#mcc687aa647\" x=\"51.378125\" y=\"38.179219\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n","      </g>\n","     </g>\n","     <g id=\"text_10\">\n","      <!-- $\\mathdefault{10^{1}}$ -->\n","      <g transform=\"translate(26.778125 41.978437)scale(0.1 -0.1)\">\n","       <use xlink:href=\"#DejaVuSans-31\" transform=\"translate(0 0.684375)\"/>\n","       <use xlink:href=\"#DejaVuSans-30\" transform=\"translate(63.623047 0.684375)\"/>\n","       <use xlink:href=\"#DejaVuSans-31\" transform=\"translate(128.203125 38.965625)scale(0.7)\"/>\n","      </g>\n","     </g>\n","    </g>\n","    <g id=\"ytick_6\">\n","     <g id=\"line2d_19\">\n","      <path d=\"M 51.378125 10.999219 \n","L 246.678125 10.999219 \n","\" clip-path=\"url(#pa12fe637e7)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n","     </g>\n","     <g id=\"line2d_20\">\n","      <g>\n","       <use xlink:href=\"#mcc687aa647\" x=\"51.378125\" y=\"10.999219\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n","      </g>\n","     </g>\n","     <g id=\"text_11\">\n","      <!-- $\\mathdefault{10^{2}}$ -->\n","      <g transform=\"translate(26.778125 14.798437)scale(0.1 -0.1)\">\n","       <use xlink:href=\"#DejaVuSans-31\" transform=\"translate(0 0.765625)\"/>\n","       <use xlink:href=\"#DejaVuSans-30\" transform=\"translate(63.623047 0.765625)\"/>\n","       <use xlink:href=\"#DejaVuSans-32\" transform=\"translate(128.203125 39.046875)scale(0.7)\"/>\n","      </g>\n","     </g>\n","    </g>\n","    <g id=\"ytick_7\">\n","     <g id=\"line2d_21\">\n","      <defs>\n","       <path id=\"m5826843031\" d=\"M 0 0 \n","L -2 0 \n","\" style=\"stroke: #000000; stroke-width: 0.6\"/>\n","      </defs>\n","      <g>\n","       <use xlink:href=\"#m5826843031\" x=\"51.378125\" y=\"138.717223\" style=\"stroke: #000000; stroke-width: 0.6\"/>\n","      </g>\n","     </g>\n","    </g>\n","    <g id=\"ytick_8\">\n","     <g id=\"line2d_22\">\n","      <g>\n","       <use xlink:href=\"#m5826843031\" x=\"51.378125\" y=\"133.931063\" style=\"stroke: #000000; stroke-width: 0.6\"/>\n","      </g>\n","     </g>\n","    </g>\n","    <g id=\"ytick_9\">\n","     <g id=\"line2d_23\">\n","      <g>\n","       <use xlink:href=\"#m5826843031\" x=\"51.378125\" y=\"130.535228\" style=\"stroke: #000000; stroke-width: 0.6\"/>\n","      </g>\n","     </g>\n","    </g>\n","    <g id=\"ytick_10\">\n","     <g id=\"line2d_24\">\n","      <g>\n","       <use xlink:href=\"#m5826843031\" x=\"51.378125\" y=\"127.901214\" style=\"stroke: #000000; stroke-width: 0.6\"/>\n","      </g>\n","     </g>\n","    </g>\n","    <g id=\"ytick_11\">\n","     <g id=\"line2d_25\">\n","      <g>\n","       <use xlink:href=\"#m5826843031\" x=\"51.378125\" y=\"125.749068\" style=\"stroke: #000000; stroke-width: 0.6\"/>\n","      </g>\n","     </g>\n","    </g>\n","    <g id=\"ytick_12\">\n","     <g id=\"line2d_26\">\n","      <g>\n","       <use xlink:href=\"#m5826843031\" x=\"51.378125\" y=\"123.929454\" style=\"stroke: #000000; stroke-width: 0.6\"/>\n","      </g>\n","     </g>\n","    </g>\n","    <g id=\"ytick_13\">\n","     <g id=\"line2d_27\">\n","      <g>\n","       <use xlink:href=\"#m5826843031\" x=\"51.378125\" y=\"122.353233\" style=\"stroke: #000000; stroke-width: 0.6\"/>\n","      </g>\n","     </g>\n","    </g>\n","    <g id=\"ytick_14\">\n","     <g id=\"line2d_28\">\n","      <g>\n","       <use xlink:href=\"#m5826843031\" x=\"51.378125\" y=\"120.962907\" style=\"stroke: #000000; stroke-width: 0.6\"/>\n","      </g>\n","     </g>\n","    </g>\n","    <g id=\"ytick_15\">\n","     <g id=\"line2d_29\">\n","      <g>\n","       <use xlink:href=\"#m5826843031\" x=\"51.378125\" y=\"111.537223\" style=\"stroke: #000000; stroke-width: 0.6\"/>\n","      </g>\n","     </g>\n","    </g>\n","    <g id=\"ytick_16\">\n","     <g id=\"line2d_30\">\n","      <g>\n","       <use xlink:href=\"#m5826843031\" x=\"51.378125\" y=\"106.751063\" style=\"stroke: #000000; stroke-width: 0.6\"/>\n","      </g>\n","     </g>\n","    </g>\n","    <g id=\"ytick_17\">\n","     <g id=\"line2d_31\">\n","      <g>\n","       <use xlink:href=\"#m5826843031\" x=\"51.378125\" y=\"103.355228\" style=\"stroke: #000000; stroke-width: 0.6\"/>\n","      </g>\n","     </g>\n","    </g>\n","    <g id=\"ytick_18\">\n","     <g id=\"line2d_32\">\n","      <g>\n","       <use xlink:href=\"#m5826843031\" x=\"51.378125\" y=\"100.721214\" style=\"stroke: #000000; stroke-width: 0.6\"/>\n","      </g>\n","     </g>\n","    </g>\n","    <g id=\"ytick_19\">\n","     <g id=\"line2d_33\">\n","      <g>\n","       <use xlink:href=\"#m5826843031\" x=\"51.378125\" y=\"98.569068\" style=\"stroke: #000000; stroke-width: 0.6\"/>\n","      </g>\n","     </g>\n","    </g>\n","    <g id=\"ytick_20\">\n","     <g id=\"line2d_34\">\n","      <g>\n","       <use xlink:href=\"#m5826843031\" x=\"51.378125\" y=\"96.749454\" style=\"stroke: #000000; stroke-width: 0.6\"/>\n","      </g>\n","     </g>\n","    </g>\n","    <g id=\"ytick_21\">\n","     <g id=\"line2d_35\">\n","      <g>\n","       <use xlink:href=\"#m5826843031\" x=\"51.378125\" y=\"95.173233\" style=\"stroke: #000000; stroke-width: 0.6\"/>\n","      </g>\n","     </g>\n","    </g>\n","    <g id=\"ytick_22\">\n","     <g id=\"line2d_36\">\n","      <g>\n","       <use xlink:href=\"#m5826843031\" x=\"51.378125\" y=\"93.782907\" style=\"stroke: #000000; stroke-width: 0.6\"/>\n","      </g>\n","     </g>\n","    </g>\n","    <g id=\"ytick_23\">\n","     <g id=\"line2d_37\">\n","      <g>\n","       <use xlink:href=\"#m5826843031\" x=\"51.378125\" y=\"84.357223\" style=\"stroke: #000000; stroke-width: 0.6\"/>\n","      </g>\n","     </g>\n","    </g>\n","    <g id=\"ytick_24\">\n","     <g id=\"line2d_38\">\n","      <g>\n","       <use xlink:href=\"#m5826843031\" x=\"51.378125\" y=\"79.571063\" style=\"stroke: #000000; stroke-width: 0.6\"/>\n","      </g>\n","     </g>\n","    </g>\n","    <g id=\"ytick_25\">\n","     <g id=\"line2d_39\">\n","      <g>\n","       <use xlink:href=\"#m5826843031\" x=\"51.378125\" y=\"76.175228\" style=\"stroke: #000000; stroke-width: 0.6\"/>\n","      </g>\n","     </g>\n","    </g>\n","    <g id=\"ytick_26\">\n","     <g id=\"line2d_40\">\n","      <g>\n","       <use xlink:href=\"#m5826843031\" x=\"51.378125\" y=\"73.541214\" style=\"stroke: #000000; stroke-width: 0.6\"/>\n","      </g>\n","     </g>\n","    </g>\n","    <g id=\"ytick_27\">\n","     <g id=\"line2d_41\">\n","      <g>\n","       <use xlink:href=\"#m5826843031\" x=\"51.378125\" y=\"71.389068\" style=\"stroke: #000000; stroke-width: 0.6\"/>\n","      </g>\n","     </g>\n","    </g>\n","    <g id=\"ytick_28\">\n","     <g id=\"line2d_42\">\n","      <g>\n","       <use xlink:href=\"#m5826843031\" x=\"51.378125\" y=\"69.569454\" style=\"stroke: #000000; stroke-width: 0.6\"/>\n","      </g>\n","     </g>\n","    </g>\n","    <g id=\"ytick_29\">\n","     <g id=\"line2d_43\">\n","      <g>\n","       <use xlink:href=\"#m5826843031\" x=\"51.378125\" y=\"67.993233\" style=\"stroke: #000000; stroke-width: 0.6\"/>\n","      </g>\n","     </g>\n","    </g>\n","    <g id=\"ytick_30\">\n","     <g id=\"line2d_44\">\n","      <g>\n","       <use xlink:href=\"#m5826843031\" x=\"51.378125\" y=\"66.602907\" style=\"stroke: #000000; stroke-width: 0.6\"/>\n","      </g>\n","     </g>\n","    </g>\n","    <g id=\"ytick_31\">\n","     <g id=\"line2d_45\">\n","      <g>\n","       <use xlink:href=\"#m5826843031\" x=\"51.378125\" y=\"57.177223\" style=\"stroke: #000000; stroke-width: 0.6\"/>\n","      </g>\n","     </g>\n","    </g>\n","    <g id=\"ytick_32\">\n","     <g id=\"line2d_46\">\n","      <g>\n","       <use xlink:href=\"#m5826843031\" x=\"51.378125\" y=\"52.391063\" style=\"stroke: #000000; stroke-width: 0.6\"/>\n","      </g>\n","     </g>\n","    </g>\n","    <g id=\"ytick_33\">\n","     <g id=\"line2d_47\">\n","      <g>\n","       <use xlink:href=\"#m5826843031\" x=\"51.378125\" y=\"48.995228\" style=\"stroke: #000000; stroke-width: 0.6\"/>\n","      </g>\n","     </g>\n","    </g>\n","    <g id=\"ytick_34\">\n","     <g id=\"line2d_48\">\n","      <g>\n","       <use xlink:href=\"#m5826843031\" x=\"51.378125\" y=\"46.361214\" style=\"stroke: #000000; stroke-width: 0.6\"/>\n","      </g>\n","     </g>\n","    </g>\n","    <g id=\"ytick_35\">\n","     <g id=\"line2d_49\">\n","      <g>\n","       <use xlink:href=\"#m5826843031\" x=\"51.378125\" y=\"44.209068\" style=\"stroke: #000000; stroke-width: 0.6\"/>\n","      </g>\n","     </g>\n","    </g>\n","    <g id=\"ytick_36\">\n","     <g id=\"line2d_50\">\n","      <g>\n","       <use xlink:href=\"#m5826843031\" x=\"51.378125\" y=\"42.389454\" style=\"stroke: #000000; stroke-width: 0.6\"/>\n","      </g>\n","     </g>\n","    </g>\n","    <g id=\"ytick_37\">\n","     <g id=\"line2d_51\">\n","      <g>\n","       <use xlink:href=\"#m5826843031\" x=\"51.378125\" y=\"40.813233\" style=\"stroke: #000000; stroke-width: 0.6\"/>\n","      </g>\n","     </g>\n","    </g>\n","    <g id=\"ytick_38\">\n","     <g id=\"line2d_52\">\n","      <g>\n","       <use xlink:href=\"#m5826843031\" x=\"51.378125\" y=\"39.422907\" style=\"stroke: #000000; stroke-width: 0.6\"/>\n","      </g>\n","     </g>\n","    </g>\n","    <g id=\"ytick_39\">\n","     <g id=\"line2d_53\">\n","      <g>\n","       <use xlink:href=\"#m5826843031\" x=\"51.378125\" y=\"29.997223\" style=\"stroke: #000000; stroke-width: 0.6\"/>\n","      </g>\n","     </g>\n","    </g>\n","    <g id=\"ytick_40\">\n","     <g id=\"line2d_54\">\n","      <g>\n","       <use xlink:href=\"#m5826843031\" x=\"51.378125\" y=\"25.211063\" style=\"stroke: #000000; stroke-width: 0.6\"/>\n","      </g>\n","     </g>\n","    </g>\n","    <g id=\"ytick_41\">\n","     <g id=\"line2d_55\">\n","      <g>\n","       <use xlink:href=\"#m5826843031\" x=\"51.378125\" y=\"21.815228\" style=\"stroke: #000000; stroke-width: 0.6\"/>\n","      </g>\n","     </g>\n","    </g>\n","    <g id=\"ytick_42\">\n","     <g id=\"line2d_56\">\n","      <g>\n","       <use xlink:href=\"#m5826843031\" x=\"51.378125\" y=\"19.181214\" style=\"stroke: #000000; stroke-width: 0.6\"/>\n","      </g>\n","     </g>\n","    </g>\n","    <g id=\"ytick_43\">\n","     <g id=\"line2d_57\">\n","      <g>\n","       <use xlink:href=\"#m5826843031\" x=\"51.378125\" y=\"17.029068\" style=\"stroke: #000000; stroke-width: 0.6\"/>\n","      </g>\n","     </g>\n","    </g>\n","    <g id=\"ytick_44\">\n","     <g id=\"line2d_58\">\n","      <g>\n","       <use xlink:href=\"#m5826843031\" x=\"51.378125\" y=\"15.209454\" style=\"stroke: #000000; stroke-width: 0.6\"/>\n","      </g>\n","     </g>\n","    </g>\n","    <g id=\"ytick_45\">\n","     <g id=\"line2d_59\">\n","      <g>\n","       <use xlink:href=\"#m5826843031\" x=\"51.378125\" y=\"13.633233\" style=\"stroke: #000000; stroke-width: 0.6\"/>\n","      </g>\n","     </g>\n","    </g>\n","    <g id=\"ytick_46\">\n","     <g id=\"line2d_60\">\n","      <g>\n","       <use xlink:href=\"#m5826843031\" x=\"51.378125\" y=\"12.242907\" style=\"stroke: #000000; stroke-width: 0.6\"/>\n","      </g>\n","     </g>\n","    </g>\n","    <g id=\"text_12\">\n","     <!-- loss -->\n","     <g transform=\"translate(14.798437 88.607031)rotate(-90)scale(0.1 -0.1)\">\n","      <defs>\n","       <path id=\"DejaVuSans-6c\" d=\"M 603 4863 \n","L 1178 4863 \n","L 1178 0 \n","L 603 0 \n","L 603 4863 \n","z\n","\" transform=\"scale(0.015625)\"/>\n","       <path id=\"DejaVuSans-73\" d=\"M 2834 3397 \n","L 2834 2853 \n","Q 2591 2978 2328 3040 \n","Q 2066 3103 1784 3103 \n","Q 1356 3103 1142 2972 \n","Q 928 2841 928 2578 \n","Q 928 2378 1081 2264 \n","Q 1234 2150 1697 2047 \n","L 1894 2003 \n","Q 2506 1872 2764 1633 \n","Q 3022 1394 3022 966 \n","Q 3022 478 2636 193 \n","Q 2250 -91 1575 -91 \n","Q 1294 -91 989 -36 \n","Q 684 19 347 128 \n","L 347 722 \n","Q 666 556 975 473 \n","Q 1284 391 1588 391 \n","Q 1994 391 2212 530 \n","Q 2431 669 2431 922 \n","Q 2431 1156 2273 1281 \n","Q 2116 1406 1581 1522 \n","L 1381 1569 \n","Q 847 1681 609 1914 \n","Q 372 2147 372 2553 \n","Q 372 3047 722 3315 \n","Q 1072 3584 1716 3584 \n","Q 2034 3584 2315 3537 \n","Q 2597 3491 2834 3397 \n","z\n","\" transform=\"scale(0.015625)\"/>\n","      </defs>\n","      <use xlink:href=\"#DejaVuSans-6c\"/>\n","      <use xlink:href=\"#DejaVuSans-6f\" x=\"27.783203\"/>\n","      <use xlink:href=\"#DejaVuSans-73\" x=\"88.964844\"/>\n","      <use xlink:href=\"#DejaVuSans-73\" x=\"141.064453\"/>\n","     </g>\n","    </g>\n","   </g>\n","   <g id=\"line2d_61\">\n","    <path d=\"M 51.378125 29.495696 \n","L 60.678125 56.573508 \n","L 70.467599 69.689101 \n","L 80.257072 81.199868 \n","L 90.046546 92.062841 \n","L 99.83602 101.836399 \n","L 109.625493 109.598216 \n","L 119.414967 114.650936 \n","L 129.204441 117.284438 \n","L 138.993914 118.446498 \n","L 148.783388 118.91446 \n","L 158.572862 119.095248 \n","L 168.362336 119.164297 \n","L 178.151809 119.190663 \n","L 187.941283 119.200575 \n","L 197.730757 119.205069 \n","L 207.52023 119.206627 \n","L 217.309704 119.20693 \n","L 227.099178 119.207412 \n","L 236.888651 119.207863 \n","L 246.678125 119.207436 \n","\" clip-path=\"url(#pa12fe637e7)\" style=\"fill: none; stroke: #1f77b4; stroke-width: 1.5; stroke-linecap: square\"/>\n","   </g>\n","   <g id=\"line2d_62\">\n","    <path d=\"M 51.378125 26.090354 \n","L 60.678125 49.028596 \n","L 70.467599 63.225054 \n","L 80.257072 75.645516 \n","L 90.046546 87.405196 \n","L 99.83602 98.520625 \n","L 109.625493 108.094319 \n","L 119.414967 114.875241 \n","L 129.204441 118.318598 \n","L 138.993914 119.53098 \n","L 148.783388 119.869033 \n","L 158.572862 119.798241 \n","L 168.362336 119.694188 \n","L 178.151809 119.608555 \n","L 187.941283 119.596537 \n","L 197.730757 119.561197 \n","L 207.52023 119.524187 \n","L 217.309704 119.465666 \n","L 227.099178 119.46717 \n","L 236.888651 119.468372 \n","L 246.678125 119.438932 \n","\" clip-path=\"url(#pa12fe637e7)\" style=\"fill: none; stroke-dasharray: 5.55,2.4; stroke-dashoffset: 0; stroke: #bf00bf; stroke-width: 1.5\"/>\n","   </g>\n","   <g id=\"patch_3\">\n","    <path d=\"M 51.378125 146.899219 \n","L 51.378125 10.999219 \n","\" style=\"fill: none; stroke: #000000; stroke-width: 0.8; stroke-linejoin: miter; stroke-linecap: square\"/>\n","   </g>\n","   <g id=\"patch_4\">\n","    <path d=\"M 246.678125 146.899219 \n","L 246.678125 10.999219 \n","\" style=\"fill: none; stroke: #000000; stroke-width: 0.8; stroke-linejoin: miter; stroke-linecap: square\"/>\n","   </g>\n","   <g id=\"patch_5\">\n","    <path d=\"M 51.378125 146.899219 \n","L 246.678125 146.899219 \n","\" style=\"fill: none; stroke: #000000; stroke-width: 0.8; stroke-linejoin: miter; stroke-linecap: square\"/>\n","   </g>\n","   <g id=\"patch_6\">\n","    <path d=\"M 51.378125 10.999219 \n","L 246.678125 10.999219 \n","\" style=\"fill: none; stroke: #000000; stroke-width: 0.8; stroke-linejoin: miter; stroke-linecap: square\"/>\n","   </g>\n","   <g id=\"legend_1\">\n","    <g id=\"patch_7\">\n","     <path d=\"M 184.403125 48.355469 \n","L 239.678125 48.355469 \n","Q 241.678125 48.355469 241.678125 46.355469 \n","L 241.678125 17.999219 \n","Q 241.678125 15.999219 239.678125 15.999219 \n","L 184.403125 15.999219 \n","Q 182.403125 15.999219 182.403125 17.999219 \n","L 182.403125 46.355469 \n","Q 182.403125 48.355469 184.403125 48.355469 \n","z\n","\" style=\"fill: #ffffff; opacity: 0.8; stroke: #cccccc; stroke-linejoin: miter\"/>\n","    </g>\n","    <g id=\"line2d_63\">\n","     <path d=\"M 186.403125 24.097656 \n","L 196.403125 24.097656 \n","L 206.403125 24.097656 \n","\" style=\"fill: none; stroke: #1f77b4; stroke-width: 1.5; stroke-linecap: square\"/>\n","    </g>\n","    <g id=\"text_13\">\n","     <!-- train -->\n","     <g transform=\"translate(214.403125 27.597656)scale(0.1 -0.1)\">\n","      <defs>\n","       <path id=\"DejaVuSans-74\" d=\"M 1172 4494 \n","L 1172 3500 \n","L 2356 3500 \n","L 2356 3053 \n","L 1172 3053 \n","L 1172 1153 \n","Q 1172 725 1289 603 \n","Q 1406 481 1766 481 \n","L 2356 481 \n","L 2356 0 \n","L 1766 0 \n","Q 1100 0 847 248 \n","Q 594 497 594 1153 \n","L 594 3053 \n","L 172 3053 \n","L 172 3500 \n","L 594 3500 \n","L 594 4494 \n","L 1172 4494 \n","z\n","\" transform=\"scale(0.015625)\"/>\n","       <path id=\"DejaVuSans-72\" d=\"M 2631 2963 \n","Q 2534 3019 2420 3045 \n","Q 2306 3072 2169 3072 \n","Q 1681 3072 1420 2755 \n","Q 1159 2438 1159 1844 \n","L 1159 0 \n","L 581 0 \n","L 581 3500 \n","L 1159 3500 \n","L 1159 2956 \n","Q 1341 3275 1631 3429 \n","Q 1922 3584 2338 3584 \n","Q 2397 3584 2469 3576 \n","Q 2541 3569 2628 3553 \n","L 2631 2963 \n","z\n","\" transform=\"scale(0.015625)\"/>\n","       <path id=\"DejaVuSans-61\" d=\"M 2194 1759 \n","Q 1497 1759 1228 1600 \n","Q 959 1441 959 1056 \n","Q 959 750 1161 570 \n","Q 1363 391 1709 391 \n","Q 2188 391 2477 730 \n","Q 2766 1069 2766 1631 \n","L 2766 1759 \n","L 2194 1759 \n","z\n","M 3341 1997 \n","L 3341 0 \n","L 2766 0 \n","L 2766 531 \n","Q 2569 213 2275 61 \n","Q 1981 -91 1556 -91 \n","Q 1019 -91 701 211 \n","Q 384 513 384 1019 \n","Q 384 1609 779 1909 \n","Q 1175 2209 1959 2209 \n","L 2766 2209 \n","L 2766 2266 \n","Q 2766 2663 2505 2880 \n","Q 2244 3097 1772 3097 \n","Q 1472 3097 1187 3025 \n","Q 903 2953 641 2809 \n","L 641 3341 \n","Q 956 3463 1253 3523 \n","Q 1550 3584 1831 3584 \n","Q 2591 3584 2966 3190 \n","Q 3341 2797 3341 1997 \n","z\n","\" transform=\"scale(0.015625)\"/>\n","       <path id=\"DejaVuSans-69\" d=\"M 603 3500 \n","L 1178 3500 \n","L 1178 0 \n","L 603 0 \n","L 603 3500 \n","z\n","M 603 4863 \n","L 1178 4863 \n","L 1178 4134 \n","L 603 4134 \n","L 603 4863 \n","z\n","\" transform=\"scale(0.015625)\"/>\n","       <path id=\"DejaVuSans-6e\" d=\"M 3513 2113 \n","L 3513 0 \n","L 2938 0 \n","L 2938 2094 \n","Q 2938 2591 2744 2837 \n","Q 2550 3084 2163 3084 \n","Q 1697 3084 1428 2787 \n","Q 1159 2491 1159 1978 \n","L 1159 0 \n","L 581 0 \n","L 581 3500 \n","L 1159 3500 \n","L 1159 2956 \n","Q 1366 3272 1645 3428 \n","Q 1925 3584 2291 3584 \n","Q 2894 3584 3203 3211 \n","Q 3513 2838 3513 2113 \n","z\n","\" transform=\"scale(0.015625)\"/>\n","      </defs>\n","      <use xlink:href=\"#DejaVuSans-74\"/>\n","      <use xlink:href=\"#DejaVuSans-72\" x=\"39.208984\"/>\n","      <use xlink:href=\"#DejaVuSans-61\" x=\"80.322266\"/>\n","      <use xlink:href=\"#DejaVuSans-69\" x=\"141.601562\"/>\n","      <use xlink:href=\"#DejaVuSans-6e\" x=\"169.384766\"/>\n","     </g>\n","    </g>\n","    <g id=\"line2d_64\">\n","     <path d=\"M 186.403125 38.775781 \n","L 196.403125 38.775781 \n","L 206.403125 38.775781 \n","\" style=\"fill: none; stroke-dasharray: 5.55,2.4; stroke-dashoffset: 0; stroke: #bf00bf; stroke-width: 1.5\"/>\n","    </g>\n","    <g id=\"text_14\">\n","     <!-- test -->\n","     <g transform=\"translate(214.403125 42.275781)scale(0.1 -0.1)\">\n","      <use xlink:href=\"#DejaVuSans-74\"/>\n","      <use xlink:href=\"#DejaVuSans-65\" x=\"39.208984\"/>\n","      <use xlink:href=\"#DejaVuSans-73\" x=\"100.732422\"/>\n","      <use xlink:href=\"#DejaVuSans-74\" x=\"152.832031\"/>\n","     </g>\n","    </g>\n","   </g>\n","  </g>\n"," </g>\n"," <defs>\n","  <clipPath id=\"pa12fe637e7\">\n","   <rect x=\"51.378125\" y=\"10.999219\" width=\"195.3\" height=\"135.9\"/>\n","  </clipPath>\n"," </defs>\n","</svg>\n"],"text/plain":["<Figure size 252x180 with 1 Axes>"]},"metadata":{"needs_background":"light"},"output_type":"display_data"}],"source":["# 从多项式特征中选择前4个维度，即1,x,x^2/2!,x^3/3!\n","train(poly_features[:n_train, :4], poly_features[n_train:, :4],\n","      labels[:n_train], labels[n_train:])"],"id":"4120f7d7"},{"cell_type":"markdown","metadata":{"origin_pos":18,"id":"57d1f833"},"source":["### [**线性函数拟合(欠拟合)**]\n","\n","让我们再看看线性函数拟合，减少该模型的训练损失相对困难。\n","在最后一个迭代周期完成后，训练损失仍然很高。\n","当用来拟合非线性模式（如这里的三阶多项式函数）时，线性模型容易欠拟合。\n"],"id":"57d1f833"},{"cell_type":"code","execution_count":null,"metadata":{"execution":{"iopub.execute_input":"2022-07-31T02:20:53.412406Z","iopub.status.busy":"2022-07-31T02:20:53.411913Z","iopub.status.idle":"2022-07-31T02:21:05.651102Z","shell.execute_reply":"2022-07-31T02:21:05.650440Z"},"origin_pos":19,"tab":["pytorch"],"id":"f2532aad","outputId":"cbcadc58-db4f-4b7f-f9bd-50d525223b78"},"outputs":[{"name":"stdout","output_type":"stream","text":["weight: [[3.4582274 3.7159562]]\n"]},{"data":{"image/svg+xml":["<?xml version=\"1.0\" encoding=\"utf-8\" standalone=\"no\"?>\n","<!DOCTYPE svg PUBLIC \"-//W3C//DTD SVG 1.1//EN\"\n","  \"http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd\">\n","<svg xmlns:xlink=\"http://www.w3.org/1999/xlink\" width=\"263.421875pt\" height=\"184.455469pt\" viewBox=\"0 0 263.421875 184.455469\" xmlns=\"http://www.w3.org/2000/svg\" version=\"1.1\">\n"," <metadata>\n","  <rdf:RDF xmlns:dc=\"http://purl.org/dc/elements/1.1/\" xmlns:cc=\"http://creativecommons.org/ns#\" xmlns:rdf=\"http://www.w3.org/1999/02/22-rdf-syntax-ns#\">\n","   <cc:Work>\n","    <dc:type rdf:resource=\"http://purl.org/dc/dcmitype/StillImage\"/>\n","    <dc:date>2022-07-31T02:21:05.522735</dc:date>\n","    <dc:format>image/svg+xml</dc:format>\n","    <dc:creator>\n","     <cc:Agent>\n","      <dc:title>Matplotlib v3.5.1, https://matplotlib.org/</dc:title>\n","     </cc:Agent>\n","    </dc:creator>\n","   </cc:Work>\n","  </rdf:RDF>\n"," </metadata>\n"," <defs>\n","  <style type=\"text/css\">*{stroke-linejoin: round; stroke-linecap: butt}</style>\n"," </defs>\n"," <g id=\"figure_1\">\n","  <g id=\"patch_1\">\n","   <path d=\"M 0 184.455469 \n","L 263.421875 184.455469 \n","L 263.421875 0 \n","L 0 0 \n","L 0 184.455469 \n","z\n","\" style=\"fill: none\"/>\n","  </g>\n","  <g id=\"axes_1\">\n","   <g id=\"patch_2\">\n","    <path d=\"M 51.378125 146.899219 \n","L 246.678125 146.899219 \n","L 246.678125 10.999219 \n","L 51.378125 10.999219 \n","z\n","\" style=\"fill: #ffffff\"/>\n","   </g>\n","   <g id=\"matplotlib.axis_1\">\n","    <g id=\"xtick_1\">\n","     <g id=\"line2d_1\">\n","      <path d=\"M 99.83602 146.899219 \n","L 99.83602 10.999219 \n","\" clip-path=\"url(#p1d7488a525)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n","     </g>\n","     <g id=\"line2d_2\">\n","      <defs>\n","       <path id=\"m09a61e7d21\" d=\"M 0 0 \n","L 0 3.5 \n","\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n","      </defs>\n","      <g>\n","       <use xlink:href=\"#m09a61e7d21\" x=\"99.83602\" y=\"146.899219\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n","      </g>\n","     </g>\n","     <g id=\"text_1\">\n","      <!-- 100 -->\n","      <g transform=\"translate(90.29227 161.497656)scale(0.1 -0.1)\">\n","       <defs>\n","        <path id=\"DejaVuSans-31\" d=\"M 794 531 \n","L 1825 531 \n","L 1825 4091 \n","L 703 3866 \n","L 703 4441 \n","L 1819 4666 \n","L 2450 4666 \n","L 2450 531 \n","L 3481 531 \n","L 3481 0 \n","L 794 0 \n","L 794 531 \n","z\n","\" transform=\"scale(0.015625)\"/>\n","        <path id=\"DejaVuSans-30\" d=\"M 2034 4250 \n","Q 1547 4250 1301 3770 \n","Q 1056 3291 1056 2328 \n","Q 1056 1369 1301 889 \n","Q 1547 409 2034 409 \n","Q 2525 409 2770 889 \n","Q 3016 1369 3016 2328 \n","Q 3016 3291 2770 3770 \n","Q 2525 4250 2034 4250 \n","z\n","M 2034 4750 \n","Q 2819 4750 3233 4129 \n","Q 3647 3509 3647 2328 \n","Q 3647 1150 3233 529 \n","Q 2819 -91 2034 -91 \n","Q 1250 -91 836 529 \n","Q 422 1150 422 2328 \n","Q 422 3509 836 4129 \n","Q 1250 4750 2034 4750 \n","z\n","\" transform=\"scale(0.015625)\"/>\n","       </defs>\n","       <use xlink:href=\"#DejaVuSans-31\"/>\n","       <use xlink:href=\"#DejaVuSans-30\" x=\"63.623047\"/>\n","       <use xlink:href=\"#DejaVuSans-30\" x=\"127.246094\"/>\n","      </g>\n","     </g>\n","    </g>\n","    <g id=\"xtick_2\">\n","     <g id=\"line2d_3\">\n","      <path d=\"M 148.783388 146.899219 \n","L 148.783388 10.999219 \n","\" clip-path=\"url(#p1d7488a525)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n","     </g>\n","     <g id=\"line2d_4\">\n","      <g>\n","       <use xlink:href=\"#m09a61e7d21\" x=\"148.783388\" y=\"146.899219\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n","      </g>\n","     </g>\n","     <g id=\"text_2\">\n","      <!-- 200 -->\n","      <g transform=\"translate(139.239638 161.497656)scale(0.1 -0.1)\">\n","       <defs>\n","        <path id=\"DejaVuSans-32\" d=\"M 1228 531 \n","L 3431 531 \n","L 3431 0 \n","L 469 0 \n","L 469 531 \n","Q 828 903 1448 1529 \n","Q 2069 2156 2228 2338 \n","Q 2531 2678 2651 2914 \n","Q 2772 3150 2772 3378 \n","Q 2772 3750 2511 3984 \n","Q 2250 4219 1831 4219 \n","Q 1534 4219 1204 4116 \n","Q 875 4013 500 3803 \n","L 500 4441 \n","Q 881 4594 1212 4672 \n","Q 1544 4750 1819 4750 \n","Q 2544 4750 2975 4387 \n","Q 3406 4025 3406 3419 \n","Q 3406 3131 3298 2873 \n","Q 3191 2616 2906 2266 \n","Q 2828 2175 2409 1742 \n","Q 1991 1309 1228 531 \n","z\n","\" transform=\"scale(0.015625)\"/>\n","       </defs>\n","       <use xlink:href=\"#DejaVuSans-32\"/>\n","       <use xlink:href=\"#DejaVuSans-30\" x=\"63.623047\"/>\n","       <use xlink:href=\"#DejaVuSans-30\" x=\"127.246094\"/>\n","      </g>\n","     </g>\n","    </g>\n","    <g id=\"xtick_3\">\n","     <g id=\"line2d_5\">\n","      <path d=\"M 197.730757 146.899219 \n","L 197.730757 10.999219 \n","\" clip-path=\"url(#p1d7488a525)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n","     </g>\n","     <g id=\"line2d_6\">\n","      <g>\n","       <use xlink:href=\"#m09a61e7d21\" x=\"197.730757\" y=\"146.899219\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n","      </g>\n","     </g>\n","     <g id=\"text_3\">\n","      <!-- 300 -->\n","      <g transform=\"translate(188.187007 161.497656)scale(0.1 -0.1)\">\n","       <defs>\n","        <path id=\"DejaVuSans-33\" d=\"M 2597 2516 \n","Q 3050 2419 3304 2112 \n","Q 3559 1806 3559 1356 \n","Q 3559 666 3084 287 \n","Q 2609 -91 1734 -91 \n","Q 1441 -91 1130 -33 \n","Q 819 25 488 141 \n","L 488 750 \n","Q 750 597 1062 519 \n","Q 1375 441 1716 441 \n","Q 2309 441 2620 675 \n","Q 2931 909 2931 1356 \n","Q 2931 1769 2642 2001 \n","Q 2353 2234 1838 2234 \n","L 1294 2234 \n","L 1294 2753 \n","L 1863 2753 \n","Q 2328 2753 2575 2939 \n","Q 2822 3125 2822 3475 \n","Q 2822 3834 2567 4026 \n","Q 2313 4219 1838 4219 \n","Q 1578 4219 1281 4162 \n","Q 984 4106 628 3988 \n","L 628 4550 \n","Q 988 4650 1302 4700 \n","Q 1616 4750 1894 4750 \n","Q 2613 4750 3031 4423 \n","Q 3450 4097 3450 3541 \n","Q 3450 3153 3228 2886 \n","Q 3006 2619 2597 2516 \n","z\n","\" transform=\"scale(0.015625)\"/>\n","       </defs>\n","       <use xlink:href=\"#DejaVuSans-33\"/>\n","       <use xlink:href=\"#DejaVuSans-30\" x=\"63.623047\"/>\n","       <use xlink:href=\"#DejaVuSans-30\" x=\"127.246094\"/>\n","      </g>\n","     </g>\n","    </g>\n","    <g id=\"xtick_4\">\n","     <g id=\"line2d_7\">\n","      <path d=\"M 246.678125 146.899219 \n","L 246.678125 10.999219 \n","\" clip-path=\"url(#p1d7488a525)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n","     </g>\n","     <g id=\"line2d_8\">\n","      <g>\n","       <use xlink:href=\"#m09a61e7d21\" x=\"246.678125\" y=\"146.899219\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n","      </g>\n","     </g>\n","     <g id=\"text_4\">\n","      <!-- 400 -->\n","      <g transform=\"translate(237.134375 161.497656)scale(0.1 -0.1)\">\n","       <defs>\n","        <path id=\"DejaVuSans-34\" d=\"M 2419 4116 \n","L 825 1625 \n","L 2419 1625 \n","L 2419 4116 \n","z\n","M 2253 4666 \n","L 3047 4666 \n","L 3047 1625 \n","L 3713 1625 \n","L 3713 1100 \n","L 3047 1100 \n","L 3047 0 \n","L 2419 0 \n","L 2419 1100 \n","L 313 1100 \n","L 313 1709 \n","L 2253 4666 \n","z\n","\" transform=\"scale(0.015625)\"/>\n","       </defs>\n","       <use xlink:href=\"#DejaVuSans-34\"/>\n","       <use xlink:href=\"#DejaVuSans-30\" x=\"63.623047\"/>\n","       <use xlink:href=\"#DejaVuSans-30\" x=\"127.246094\"/>\n","      </g>\n","     </g>\n","    </g>\n","    <g id=\"text_5\">\n","     <!-- epoch -->\n","     <g transform=\"translate(133.8 175.175781)scale(0.1 -0.1)\">\n","      <defs>\n","       <path id=\"DejaVuSans-65\" d=\"M 3597 1894 \n","L 3597 1613 \n","L 953 1613 \n","Q 991 1019 1311 708 \n","Q 1631 397 2203 397 \n","Q 2534 397 2845 478 \n","Q 3156 559 3463 722 \n","L 3463 178 \n","Q 3153 47 2828 -22 \n","Q 2503 -91 2169 -91 \n","Q 1331 -91 842 396 \n","Q 353 884 353 1716 \n","Q 353 2575 817 3079 \n","Q 1281 3584 2069 3584 \n","Q 2775 3584 3186 3129 \n","Q 3597 2675 3597 1894 \n","z\n","M 3022 2063 \n","Q 3016 2534 2758 2815 \n","Q 2500 3097 2075 3097 \n","Q 1594 3097 1305 2825 \n","Q 1016 2553 972 2059 \n","L 3022 2063 \n","z\n","\" transform=\"scale(0.015625)\"/>\n","       <path id=\"DejaVuSans-70\" d=\"M 1159 525 \n","L 1159 -1331 \n","L 581 -1331 \n","L 581 3500 \n","L 1159 3500 \n","L 1159 2969 \n","Q 1341 3281 1617 3432 \n","Q 1894 3584 2278 3584 \n","Q 2916 3584 3314 3078 \n","Q 3713 2572 3713 1747 \n","Q 3713 922 3314 415 \n","Q 2916 -91 2278 -91 \n","Q 1894 -91 1617 61 \n","Q 1341 213 1159 525 \n","z\n","M 3116 1747 \n","Q 3116 2381 2855 2742 \n","Q 2594 3103 2138 3103 \n","Q 1681 3103 1420 2742 \n","Q 1159 2381 1159 1747 \n","Q 1159 1113 1420 752 \n","Q 1681 391 2138 391 \n","Q 2594 391 2855 752 \n","Q 3116 1113 3116 1747 \n","z\n","\" transform=\"scale(0.015625)\"/>\n","       <path id=\"DejaVuSans-6f\" d=\"M 1959 3097 \n","Q 1497 3097 1228 2736 \n","Q 959 2375 959 1747 \n","Q 959 1119 1226 758 \n","Q 1494 397 1959 397 \n","Q 2419 397 2687 759 \n","Q 2956 1122 2956 1747 \n","Q 2956 2369 2687 2733 \n","Q 2419 3097 1959 3097 \n","z\n","M 1959 3584 \n","Q 2709 3584 3137 3096 \n","Q 3566 2609 3566 1747 \n","Q 3566 888 3137 398 \n","Q 2709 -91 1959 -91 \n","Q 1206 -91 779 398 \n","Q 353 888 353 1747 \n","Q 353 2609 779 3096 \n","Q 1206 3584 1959 3584 \n","z\n","\" transform=\"scale(0.015625)\"/>\n","       <path id=\"DejaVuSans-63\" d=\"M 3122 3366 \n","L 3122 2828 \n","Q 2878 2963 2633 3030 \n","Q 2388 3097 2138 3097 \n","Q 1578 3097 1268 2742 \n","Q 959 2388 959 1747 \n","Q 959 1106 1268 751 \n","Q 1578 397 2138 397 \n","Q 2388 397 2633 464 \n","Q 2878 531 3122 666 \n","L 3122 134 \n","Q 2881 22 2623 -34 \n","Q 2366 -91 2075 -91 \n","Q 1284 -91 818 406 \n","Q 353 903 353 1747 \n","Q 353 2603 823 3093 \n","Q 1294 3584 2113 3584 \n","Q 2378 3584 2631 3529 \n","Q 2884 3475 3122 3366 \n","z\n","\" transform=\"scale(0.015625)\"/>\n","       <path id=\"DejaVuSans-68\" d=\"M 3513 2113 \n","L 3513 0 \n","L 2938 0 \n","L 2938 2094 \n","Q 2938 2591 2744 2837 \n","Q 2550 3084 2163 3084 \n","Q 1697 3084 1428 2787 \n","Q 1159 2491 1159 1978 \n","L 1159 0 \n","L 581 0 \n","L 581 4863 \n","L 1159 4863 \n","L 1159 2956 \n","Q 1366 3272 1645 3428 \n","Q 1925 3584 2291 3584 \n","Q 2894 3584 3203 3211 \n","Q 3513 2838 3513 2113 \n","z\n","\" transform=\"scale(0.015625)\"/>\n","      </defs>\n","      <use xlink:href=\"#DejaVuSans-65\"/>\n","      <use xlink:href=\"#DejaVuSans-70\" x=\"61.523438\"/>\n","      <use xlink:href=\"#DejaVuSans-6f\" x=\"125\"/>\n","      <use xlink:href=\"#DejaVuSans-63\" x=\"186.181641\"/>\n","      <use xlink:href=\"#DejaVuSans-68\" x=\"241.162109\"/>\n","     </g>\n","    </g>\n","   </g>\n","   <g id=\"matplotlib.axis_2\">\n","    <g id=\"ytick_1\">\n","     <g id=\"line2d_9\">\n","      <path d=\"M 51.378125 146.899219 \n","L 246.678125 146.899219 \n","\" clip-path=\"url(#p1d7488a525)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n","     </g>\n","     <g id=\"line2d_10\">\n","      <defs>\n","       <path id=\"mf91d827d95\" d=\"M 0 0 \n","L -3.5 0 \n","\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n","      </defs>\n","      <g>\n","       <use xlink:href=\"#mf91d827d95\" x=\"51.378125\" y=\"146.899219\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n","      </g>\n","     </g>\n","     <g id=\"text_6\">\n","      <!-- $\\mathdefault{10^{-3}}$ -->\n","      <g transform=\"translate(20.878125 150.698438)scale(0.1 -0.1)\">\n","       <defs>\n","        <path id=\"DejaVuSans-2212\" d=\"M 678 2272 \n","L 4684 2272 \n","L 4684 1741 \n","L 678 1741 \n","L 678 2272 \n","z\n","\" transform=\"scale(0.015625)\"/>\n","       </defs>\n","       <use xlink:href=\"#DejaVuSans-31\" transform=\"translate(0 0.765625)\"/>\n","       <use xlink:href=\"#DejaVuSans-30\" transform=\"translate(63.623047 0.765625)\"/>\n","       <use xlink:href=\"#DejaVuSans-2212\" transform=\"translate(128.203125 39.046875)scale(0.7)\"/>\n","       <use xlink:href=\"#DejaVuSans-33\" transform=\"translate(186.855469 39.046875)scale(0.7)\"/>\n","      </g>\n","     </g>\n","    </g>\n","    <g id=\"ytick_2\">\n","     <g id=\"line2d_11\">\n","      <path d=\"M 51.378125 119.719219 \n","L 246.678125 119.719219 \n","\" clip-path=\"url(#p1d7488a525)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n","     </g>\n","     <g id=\"line2d_12\">\n","      <g>\n","       <use xlink:href=\"#mf91d827d95\" x=\"51.378125\" y=\"119.719219\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n","      </g>\n","     </g>\n","     <g id=\"text_7\">\n","      <!-- $\\mathdefault{10^{-2}}$ -->\n","      <g transform=\"translate(20.878125 123.518438)scale(0.1 -0.1)\">\n","       <use xlink:href=\"#DejaVuSans-31\" transform=\"translate(0 0.765625)\"/>\n","       <use xlink:href=\"#DejaVuSans-30\" transform=\"translate(63.623047 0.765625)\"/>\n","       <use xlink:href=\"#DejaVuSans-2212\" transform=\"translate(128.203125 39.046875)scale(0.7)\"/>\n","       <use xlink:href=\"#DejaVuSans-32\" transform=\"translate(186.855469 39.046875)scale(0.7)\"/>\n","      </g>\n","     </g>\n","    </g>\n","    <g id=\"ytick_3\">\n","     <g id=\"line2d_13\">\n","      <path d=\"M 51.378125 92.539219 \n","L 246.678125 92.539219 \n","\" clip-path=\"url(#p1d7488a525)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n","     </g>\n","     <g id=\"line2d_14\">\n","      <g>\n","       <use xlink:href=\"#mf91d827d95\" x=\"51.378125\" y=\"92.539219\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n","      </g>\n","     </g>\n","     <g id=\"text_8\">\n","      <!-- $\\mathdefault{10^{-1}}$ -->\n","      <g transform=\"translate(20.878125 96.338437)scale(0.1 -0.1)\">\n","       <use xlink:href=\"#DejaVuSans-31\" transform=\"translate(0 0.684375)\"/>\n","       <use xlink:href=\"#DejaVuSans-30\" transform=\"translate(63.623047 0.684375)\"/>\n","       <use xlink:href=\"#DejaVuSans-2212\" transform=\"translate(128.203125 38.965625)scale(0.7)\"/>\n","       <use xlink:href=\"#DejaVuSans-31\" transform=\"translate(186.855469 38.965625)scale(0.7)\"/>\n","      </g>\n","     </g>\n","    </g>\n","    <g id=\"ytick_4\">\n","     <g id=\"line2d_15\">\n","      <path d=\"M 51.378125 65.359219 \n","L 246.678125 65.359219 \n","\" clip-path=\"url(#p1d7488a525)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n","     </g>\n","     <g id=\"line2d_16\">\n","      <g>\n","       <use xlink:href=\"#mf91d827d95\" x=\"51.378125\" y=\"65.359219\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n","      </g>\n","     </g>\n","     <g id=\"text_9\">\n","      <!-- $\\mathdefault{10^{0}}$ -->\n","      <g transform=\"translate(26.778125 69.158437)scale(0.1 -0.1)\">\n","       <use xlink:href=\"#DejaVuSans-31\" transform=\"translate(0 0.765625)\"/>\n","       <use xlink:href=\"#DejaVuSans-30\" transform=\"translate(63.623047 0.765625)\"/>\n","       <use xlink:href=\"#DejaVuSans-30\" transform=\"translate(128.203125 39.046875)scale(0.7)\"/>\n","      </g>\n","     </g>\n","    </g>\n","    <g id=\"ytick_5\">\n","     <g id=\"line2d_17\">\n","      <path d=\"M 51.378125 38.179219 \n","L 246.678125 38.179219 \n","\" clip-path=\"url(#p1d7488a525)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n","     </g>\n","     <g id=\"line2d_18\">\n","      <g>\n","       <use xlink:href=\"#mf91d827d95\" x=\"51.378125\" y=\"38.179219\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n","      </g>\n","     </g>\n","     <g id=\"text_10\">\n","      <!-- $\\mathdefault{10^{1}}$ -->\n","      <g transform=\"translate(26.778125 41.978437)scale(0.1 -0.1)\">\n","       <use xlink:href=\"#DejaVuSans-31\" transform=\"translate(0 0.684375)\"/>\n","       <use xlink:href=\"#DejaVuSans-30\" transform=\"translate(63.623047 0.684375)\"/>\n","       <use xlink:href=\"#DejaVuSans-31\" transform=\"translate(128.203125 38.965625)scale(0.7)\"/>\n","      </g>\n","     </g>\n","    </g>\n","    <g id=\"ytick_6\">\n","     <g id=\"line2d_19\">\n","      <path d=\"M 51.378125 10.999219 \n","L 246.678125 10.999219 \n","\" clip-path=\"url(#p1d7488a525)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n","     </g>\n","     <g id=\"line2d_20\">\n","      <g>\n","       <use xlink:href=\"#mf91d827d95\" x=\"51.378125\" y=\"10.999219\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n","      </g>\n","     </g>\n","     <g id=\"text_11\">\n","      <!-- $\\mathdefault{10^{2}}$ -->\n","      <g transform=\"translate(26.778125 14.798437)scale(0.1 -0.1)\">\n","       <use xlink:href=\"#DejaVuSans-31\" transform=\"translate(0 0.765625)\"/>\n","       <use xlink:href=\"#DejaVuSans-30\" transform=\"translate(63.623047 0.765625)\"/>\n","       <use xlink:href=\"#DejaVuSans-32\" transform=\"translate(128.203125 39.046875)scale(0.7)\"/>\n","      </g>\n","     </g>\n","    </g>\n","    <g id=\"ytick_7\">\n","     <g id=\"line2d_21\">\n","      <defs>\n","       <path id=\"me3542cfa32\" d=\"M 0 0 \n","L -2 0 \n","\" style=\"stroke: #000000; stroke-width: 0.6\"/>\n","      </defs>\n","      <g>\n","       <use xlink:href=\"#me3542cfa32\" x=\"51.378125\" y=\"138.717223\" style=\"stroke: #000000; stroke-width: 0.6\"/>\n","      </g>\n","     </g>\n","    </g>\n","    <g id=\"ytick_8\">\n","     <g id=\"line2d_22\">\n","      <g>\n","       <use xlink:href=\"#me3542cfa32\" x=\"51.378125\" y=\"133.931063\" style=\"stroke: #000000; stroke-width: 0.6\"/>\n","      </g>\n","     </g>\n","    </g>\n","    <g id=\"ytick_9\">\n","     <g id=\"line2d_23\">\n","      <g>\n","       <use xlink:href=\"#me3542cfa32\" x=\"51.378125\" y=\"130.535228\" style=\"stroke: #000000; stroke-width: 0.6\"/>\n","      </g>\n","     </g>\n","    </g>\n","    <g id=\"ytick_10\">\n","     <g id=\"line2d_24\">\n","      <g>\n","       <use xlink:href=\"#me3542cfa32\" x=\"51.378125\" y=\"127.901214\" style=\"stroke: #000000; stroke-width: 0.6\"/>\n","      </g>\n","     </g>\n","    </g>\n","    <g id=\"ytick_11\">\n","     <g id=\"line2d_25\">\n","      <g>\n","       <use xlink:href=\"#me3542cfa32\" x=\"51.378125\" y=\"125.749068\" style=\"stroke: #000000; stroke-width: 0.6\"/>\n","      </g>\n","     </g>\n","    </g>\n","    <g id=\"ytick_12\">\n","     <g id=\"line2d_26\">\n","      <g>\n","       <use xlink:href=\"#me3542cfa32\" x=\"51.378125\" y=\"123.929454\" style=\"stroke: #000000; stroke-width: 0.6\"/>\n","      </g>\n","     </g>\n","    </g>\n","    <g id=\"ytick_13\">\n","     <g id=\"line2d_27\">\n","      <g>\n","       <use xlink:href=\"#me3542cfa32\" x=\"51.378125\" y=\"122.353233\" style=\"stroke: #000000; stroke-width: 0.6\"/>\n","      </g>\n","     </g>\n","    </g>\n","    <g id=\"ytick_14\">\n","     <g id=\"line2d_28\">\n","      <g>\n","       <use xlink:href=\"#me3542cfa32\" x=\"51.378125\" y=\"120.962907\" style=\"stroke: #000000; stroke-width: 0.6\"/>\n","      </g>\n","     </g>\n","    </g>\n","    <g id=\"ytick_15\">\n","     <g id=\"line2d_29\">\n","      <g>\n","       <use xlink:href=\"#me3542cfa32\" x=\"51.378125\" y=\"111.537223\" style=\"stroke: #000000; stroke-width: 0.6\"/>\n","      </g>\n","     </g>\n","    </g>\n","    <g id=\"ytick_16\">\n","     <g id=\"line2d_30\">\n","      <g>\n","       <use xlink:href=\"#me3542cfa32\" x=\"51.378125\" y=\"106.751063\" style=\"stroke: #000000; stroke-width: 0.6\"/>\n","      </g>\n","     </g>\n","    </g>\n","    <g id=\"ytick_17\">\n","     <g id=\"line2d_31\">\n","      <g>\n","       <use xlink:href=\"#me3542cfa32\" x=\"51.378125\" y=\"103.355228\" style=\"stroke: #000000; stroke-width: 0.6\"/>\n","      </g>\n","     </g>\n","    </g>\n","    <g id=\"ytick_18\">\n","     <g id=\"line2d_32\">\n","      <g>\n","       <use xlink:href=\"#me3542cfa32\" x=\"51.378125\" y=\"100.721214\" style=\"stroke: #000000; stroke-width: 0.6\"/>\n","      </g>\n","     </g>\n","    </g>\n","    <g id=\"ytick_19\">\n","     <g id=\"line2d_33\">\n","      <g>\n","       <use xlink:href=\"#me3542cfa32\" x=\"51.378125\" y=\"98.569068\" style=\"stroke: #000000; stroke-width: 0.6\"/>\n","      </g>\n","     </g>\n","    </g>\n","    <g id=\"ytick_20\">\n","     <g id=\"line2d_34\">\n","      <g>\n","       <use xlink:href=\"#me3542cfa32\" x=\"51.378125\" y=\"96.749454\" style=\"stroke: #000000; stroke-width: 0.6\"/>\n","      </g>\n","     </g>\n","    </g>\n","    <g id=\"ytick_21\">\n","     <g id=\"line2d_35\">\n","      <g>\n","       <use xlink:href=\"#me3542cfa32\" x=\"51.378125\" y=\"95.173233\" style=\"stroke: #000000; stroke-width: 0.6\"/>\n","      </g>\n","     </g>\n","    </g>\n","    <g id=\"ytick_22\">\n","     <g id=\"line2d_36\">\n","      <g>\n","       <use xlink:href=\"#me3542cfa32\" x=\"51.378125\" y=\"93.782907\" style=\"stroke: #000000; stroke-width: 0.6\"/>\n","      </g>\n","     </g>\n","    </g>\n","    <g id=\"ytick_23\">\n","     <g id=\"line2d_37\">\n","      <g>\n","       <use xlink:href=\"#me3542cfa32\" x=\"51.378125\" y=\"84.357223\" style=\"stroke: #000000; stroke-width: 0.6\"/>\n","      </g>\n","     </g>\n","    </g>\n","    <g id=\"ytick_24\">\n","     <g id=\"line2d_38\">\n","      <g>\n","       <use xlink:href=\"#me3542cfa32\" x=\"51.378125\" y=\"79.571063\" style=\"stroke: #000000; stroke-width: 0.6\"/>\n","      </g>\n","     </g>\n","    </g>\n","    <g id=\"ytick_25\">\n","     <g id=\"line2d_39\">\n","      <g>\n","       <use xlink:href=\"#me3542cfa32\" x=\"51.378125\" y=\"76.175228\" style=\"stroke: #000000; stroke-width: 0.6\"/>\n","      </g>\n","     </g>\n","    </g>\n","    <g id=\"ytick_26\">\n","     <g id=\"line2d_40\">\n","      <g>\n","       <use xlink:href=\"#me3542cfa32\" x=\"51.378125\" y=\"73.541214\" style=\"stroke: #000000; stroke-width: 0.6\"/>\n","      </g>\n","     </g>\n","    </g>\n","    <g id=\"ytick_27\">\n","     <g id=\"line2d_41\">\n","      <g>\n","       <use xlink:href=\"#me3542cfa32\" x=\"51.378125\" y=\"71.389068\" style=\"stroke: #000000; stroke-width: 0.6\"/>\n","      </g>\n","     </g>\n","    </g>\n","    <g id=\"ytick_28\">\n","     <g id=\"line2d_42\">\n","      <g>\n","       <use xlink:href=\"#me3542cfa32\" x=\"51.378125\" y=\"69.569454\" style=\"stroke: #000000; stroke-width: 0.6\"/>\n","      </g>\n","     </g>\n","    </g>\n","    <g id=\"ytick_29\">\n","     <g id=\"line2d_43\">\n","      <g>\n","       <use xlink:href=\"#me3542cfa32\" x=\"51.378125\" y=\"67.993233\" style=\"stroke: #000000; stroke-width: 0.6\"/>\n","      </g>\n","     </g>\n","    </g>\n","    <g id=\"ytick_30\">\n","     <g id=\"line2d_44\">\n","      <g>\n","       <use xlink:href=\"#me3542cfa32\" x=\"51.378125\" y=\"66.602907\" style=\"stroke: #000000; stroke-width: 0.6\"/>\n","      </g>\n","     </g>\n","    </g>\n","    <g id=\"ytick_31\">\n","     <g id=\"line2d_45\">\n","      <g>\n","       <use xlink:href=\"#me3542cfa32\" x=\"51.378125\" y=\"57.177223\" style=\"stroke: #000000; stroke-width: 0.6\"/>\n","      </g>\n","     </g>\n","    </g>\n","    <g id=\"ytick_32\">\n","     <g id=\"line2d_46\">\n","      <g>\n","       <use xlink:href=\"#me3542cfa32\" x=\"51.378125\" y=\"52.391063\" style=\"stroke: #000000; stroke-width: 0.6\"/>\n","      </g>\n","     </g>\n","    </g>\n","    <g id=\"ytick_33\">\n","     <g id=\"line2d_47\">\n","      <g>\n","       <use xlink:href=\"#me3542cfa32\" x=\"51.378125\" y=\"48.995228\" style=\"stroke: #000000; stroke-width: 0.6\"/>\n","      </g>\n","     </g>\n","    </g>\n","    <g id=\"ytick_34\">\n","     <g id=\"line2d_48\">\n","      <g>\n","       <use xlink:href=\"#me3542cfa32\" x=\"51.378125\" y=\"46.361214\" style=\"stroke: #000000; stroke-width: 0.6\"/>\n","      </g>\n","     </g>\n","    </g>\n","    <g id=\"ytick_35\">\n","     <g id=\"line2d_49\">\n","      <g>\n","       <use xlink:href=\"#me3542cfa32\" x=\"51.378125\" y=\"44.209068\" style=\"stroke: #000000; stroke-width: 0.6\"/>\n","      </g>\n","     </g>\n","    </g>\n","    <g id=\"ytick_36\">\n","     <g id=\"line2d_50\">\n","      <g>\n","       <use xlink:href=\"#me3542cfa32\" x=\"51.378125\" y=\"42.389454\" style=\"stroke: #000000; stroke-width: 0.6\"/>\n","      </g>\n","     </g>\n","    </g>\n","    <g id=\"ytick_37\">\n","     <g id=\"line2d_51\">\n","      <g>\n","       <use xlink:href=\"#me3542cfa32\" x=\"51.378125\" y=\"40.813233\" style=\"stroke: #000000; stroke-width: 0.6\"/>\n","      </g>\n","     </g>\n","    </g>\n","    <g id=\"ytick_38\">\n","     <g id=\"line2d_52\">\n","      <g>\n","       <use xlink:href=\"#me3542cfa32\" x=\"51.378125\" y=\"39.422907\" style=\"stroke: #000000; stroke-width: 0.6\"/>\n","      </g>\n","     </g>\n","    </g>\n","    <g id=\"ytick_39\">\n","     <g id=\"line2d_53\">\n","      <g>\n","       <use xlink:href=\"#me3542cfa32\" x=\"51.378125\" y=\"29.997223\" style=\"stroke: #000000; stroke-width: 0.6\"/>\n","      </g>\n","     </g>\n","    </g>\n","    <g id=\"ytick_40\">\n","     <g id=\"line2d_54\">\n","      <g>\n","       <use xlink:href=\"#me3542cfa32\" x=\"51.378125\" y=\"25.211063\" style=\"stroke: #000000; stroke-width: 0.6\"/>\n","      </g>\n","     </g>\n","    </g>\n","    <g id=\"ytick_41\">\n","     <g id=\"line2d_55\">\n","      <g>\n","       <use xlink:href=\"#me3542cfa32\" x=\"51.378125\" y=\"21.815228\" style=\"stroke: #000000; stroke-width: 0.6\"/>\n","      </g>\n","     </g>\n","    </g>\n","    <g id=\"ytick_42\">\n","     <g id=\"line2d_56\">\n","      <g>\n","       <use xlink:href=\"#me3542cfa32\" x=\"51.378125\" y=\"19.181214\" style=\"stroke: #000000; stroke-width: 0.6\"/>\n","      </g>\n","     </g>\n","    </g>\n","    <g id=\"ytick_43\">\n","     <g id=\"line2d_57\">\n","      <g>\n","       <use xlink:href=\"#me3542cfa32\" x=\"51.378125\" y=\"17.029068\" style=\"stroke: #000000; stroke-width: 0.6\"/>\n","      </g>\n","     </g>\n","    </g>\n","    <g id=\"ytick_44\">\n","     <g id=\"line2d_58\">\n","      <g>\n","       <use xlink:href=\"#me3542cfa32\" x=\"51.378125\" y=\"15.209454\" style=\"stroke: #000000; stroke-width: 0.6\"/>\n","      </g>\n","     </g>\n","    </g>\n","    <g id=\"ytick_45\">\n","     <g id=\"line2d_59\">\n","      <g>\n","       <use xlink:href=\"#me3542cfa32\" x=\"51.378125\" y=\"13.633233\" style=\"stroke: #000000; stroke-width: 0.6\"/>\n","      </g>\n","     </g>\n","    </g>\n","    <g id=\"ytick_46\">\n","     <g id=\"line2d_60\">\n","      <g>\n","       <use xlink:href=\"#me3542cfa32\" x=\"51.378125\" y=\"12.242907\" style=\"stroke: #000000; stroke-width: 0.6\"/>\n","      </g>\n","     </g>\n","    </g>\n","    <g id=\"text_12\">\n","     <!-- loss -->\n","     <g transform=\"translate(14.798437 88.607031)rotate(-90)scale(0.1 -0.1)\">\n","      <defs>\n","       <path id=\"DejaVuSans-6c\" d=\"M 603 4863 \n","L 1178 4863 \n","L 1178 0 \n","L 603 0 \n","L 603 4863 \n","z\n","\" transform=\"scale(0.015625)\"/>\n","       <path id=\"DejaVuSans-73\" d=\"M 2834 3397 \n","L 2834 2853 \n","Q 2591 2978 2328 3040 \n","Q 2066 3103 1784 3103 \n","Q 1356 3103 1142 2972 \n","Q 928 2841 928 2578 \n","Q 928 2378 1081 2264 \n","Q 1234 2150 1697 2047 \n","L 1894 2003 \n","Q 2506 1872 2764 1633 \n","Q 3022 1394 3022 966 \n","Q 3022 478 2636 193 \n","Q 2250 -91 1575 -91 \n","Q 1294 -91 989 -36 \n","Q 684 19 347 128 \n","L 347 722 \n","Q 666 556 975 473 \n","Q 1284 391 1588 391 \n","Q 1994 391 2212 530 \n","Q 2431 669 2431 922 \n","Q 2431 1156 2273 1281 \n","Q 2116 1406 1581 1522 \n","L 1381 1569 \n","Q 847 1681 609 1914 \n","Q 372 2147 372 2553 \n","Q 372 3047 722 3315 \n","Q 1072 3584 1716 3584 \n","Q 2034 3584 2315 3537 \n","Q 2597 3491 2834 3397 \n","z\n","\" transform=\"scale(0.015625)\"/>\n","      </defs>\n","      <use xlink:href=\"#DejaVuSans-6c\"/>\n","      <use xlink:href=\"#DejaVuSans-6f\" x=\"27.783203\"/>\n","      <use xlink:href=\"#DejaVuSans-73\" x=\"88.964844\"/>\n","      <use xlink:href=\"#DejaVuSans-73\" x=\"141.064453\"/>\n","     </g>\n","    </g>\n","   </g>\n","   <g id=\"line2d_61\">\n","    <path d=\"M 51.378125 28.259852 \n","L 60.678125 42.936083 \n","L 70.467599 42.94958 \n","L 80.257072 42.950179 \n","L 90.046546 42.950009 \n","L 99.83602 42.9501 \n","L 109.625493 42.949866 \n","L 119.414967 42.949975 \n","L 129.204441 42.949659 \n","L 138.993914 42.950153 \n","L 148.783388 42.950102 \n","L 158.572862 42.949379 \n","L 168.362336 42.950172 \n","L 178.151809 42.949992 \n","L 187.941283 42.949956 \n","L 197.730757 42.94991 \n","L 207.52023 42.950106 \n","L 217.309704 42.950028 \n","L 227.099178 42.950139 \n","L 236.888651 42.950155 \n","L 246.678125 42.95016 \n","\" clip-path=\"url(#p1d7488a525)\" style=\"fill: none; stroke: #1f77b4; stroke-width: 1.5; stroke-linecap: square\"/>\n","   </g>\n","   <g id=\"line2d_62\">\n","    <path d=\"M 51.378125 24.552963 \n","L 60.678125 34.721951 \n","L 70.467599 34.78302 \n","L 80.257072 34.754686 \n","L 90.046546 34.769943 \n","L 99.83602 34.747301 \n","L 109.625493 34.768796 \n","L 119.414967 34.759333 \n","L 129.204441 34.778639 \n","L 138.993914 34.761334 \n","L 148.783388 34.74511 \n","L 158.572862 34.772098 \n","L 168.362336 34.752978 \n","L 178.151809 34.73942 \n","L 187.941283 34.769529 \n","L 197.730757 34.735595 \n","L 207.52023 34.751662 \n","L 217.309704 34.768413 \n","L 227.099178 34.746696 \n","L 236.888651 34.751241 \n","L 246.678125 34.756492 \n","\" clip-path=\"url(#p1d7488a525)\" style=\"fill: none; stroke-dasharray: 5.55,2.4; stroke-dashoffset: 0; stroke: #bf00bf; stroke-width: 1.5\"/>\n","   </g>\n","   <g id=\"patch_3\">\n","    <path d=\"M 51.378125 146.899219 \n","L 51.378125 10.999219 \n","\" style=\"fill: none; stroke: #000000; stroke-width: 0.8; stroke-linejoin: miter; stroke-linecap: square\"/>\n","   </g>\n","   <g id=\"patch_4\">\n","    <path d=\"M 246.678125 146.899219 \n","L 246.678125 10.999219 \n","\" style=\"fill: none; stroke: #000000; stroke-width: 0.8; stroke-linejoin: miter; stroke-linecap: square\"/>\n","   </g>\n","   <g id=\"patch_5\">\n","    <path d=\"M 51.378125 146.899219 \n","L 246.678125 146.899219 \n","\" style=\"fill: none; stroke: #000000; stroke-width: 0.8; stroke-linejoin: miter; stroke-linecap: square\"/>\n","   </g>\n","   <g id=\"patch_6\">\n","    <path d=\"M 51.378125 10.999219 \n","L 246.678125 10.999219 \n","\" style=\"fill: none; stroke: #000000; stroke-width: 0.8; stroke-linejoin: miter; stroke-linecap: square\"/>\n","   </g>\n","   <g id=\"legend_1\">\n","    <g id=\"patch_7\">\n","     <path d=\"M 58.378125 141.899219 \n","L 113.653125 141.899219 \n","Q 115.653125 141.899219 115.653125 139.899219 \n","L 115.653125 111.542969 \n","Q 115.653125 109.542969 113.653125 109.542969 \n","L 58.378125 109.542969 \n","Q 56.378125 109.542969 56.378125 111.542969 \n","L 56.378125 139.899219 \n","Q 56.378125 141.899219 58.378125 141.899219 \n","z\n","\" style=\"fill: #ffffff; opacity: 0.8; stroke: #cccccc; stroke-linejoin: miter\"/>\n","    </g>\n","    <g id=\"line2d_63\">\n","     <path d=\"M 60.378125 117.641406 \n","L 70.378125 117.641406 \n","L 80.378125 117.641406 \n","\" style=\"fill: none; stroke: #1f77b4; stroke-width: 1.5; stroke-linecap: square\"/>\n","    </g>\n","    <g id=\"text_13\">\n","     <!-- train -->\n","     <g transform=\"translate(88.378125 121.141406)scale(0.1 -0.1)\">\n","      <defs>\n","       <path id=\"DejaVuSans-74\" d=\"M 1172 4494 \n","L 1172 3500 \n","L 2356 3500 \n","L 2356 3053 \n","L 1172 3053 \n","L 1172 1153 \n","Q 1172 725 1289 603 \n","Q 1406 481 1766 481 \n","L 2356 481 \n","L 2356 0 \n","L 1766 0 \n","Q 1100 0 847 248 \n","Q 594 497 594 1153 \n","L 594 3053 \n","L 172 3053 \n","L 172 3500 \n","L 594 3500 \n","L 594 4494 \n","L 1172 4494 \n","z\n","\" transform=\"scale(0.015625)\"/>\n","       <path id=\"DejaVuSans-72\" d=\"M 2631 2963 \n","Q 2534 3019 2420 3045 \n","Q 2306 3072 2169 3072 \n","Q 1681 3072 1420 2755 \n","Q 1159 2438 1159 1844 \n","L 1159 0 \n","L 581 0 \n","L 581 3500 \n","L 1159 3500 \n","L 1159 2956 \n","Q 1341 3275 1631 3429 \n","Q 1922 3584 2338 3584 \n","Q 2397 3584 2469 3576 \n","Q 2541 3569 2628 3553 \n","L 2631 2963 \n","z\n","\" transform=\"scale(0.015625)\"/>\n","       <path id=\"DejaVuSans-61\" d=\"M 2194 1759 \n","Q 1497 1759 1228 1600 \n","Q 959 1441 959 1056 \n","Q 959 750 1161 570 \n","Q 1363 391 1709 391 \n","Q 2188 391 2477 730 \n","Q 2766 1069 2766 1631 \n","L 2766 1759 \n","L 2194 1759 \n","z\n","M 3341 1997 \n","L 3341 0 \n","L 2766 0 \n","L 2766 531 \n","Q 2569 213 2275 61 \n","Q 1981 -91 1556 -91 \n","Q 1019 -91 701 211 \n","Q 384 513 384 1019 \n","Q 384 1609 779 1909 \n","Q 1175 2209 1959 2209 \n","L 2766 2209 \n","L 2766 2266 \n","Q 2766 2663 2505 2880 \n","Q 2244 3097 1772 3097 \n","Q 1472 3097 1187 3025 \n","Q 903 2953 641 2809 \n","L 641 3341 \n","Q 956 3463 1253 3523 \n","Q 1550 3584 1831 3584 \n","Q 2591 3584 2966 3190 \n","Q 3341 2797 3341 1997 \n","z\n","\" transform=\"scale(0.015625)\"/>\n","       <path id=\"DejaVuSans-69\" d=\"M 603 3500 \n","L 1178 3500 \n","L 1178 0 \n","L 603 0 \n","L 603 3500 \n","z\n","M 603 4863 \n","L 1178 4863 \n","L 1178 4134 \n","L 603 4134 \n","L 603 4863 \n","z\n","\" transform=\"scale(0.015625)\"/>\n","       <path id=\"DejaVuSans-6e\" d=\"M 3513 2113 \n","L 3513 0 \n","L 2938 0 \n","L 2938 2094 \n","Q 2938 2591 2744 2837 \n","Q 2550 3084 2163 3084 \n","Q 1697 3084 1428 2787 \n","Q 1159 2491 1159 1978 \n","L 1159 0 \n","L 581 0 \n","L 581 3500 \n","L 1159 3500 \n","L 1159 2956 \n","Q 1366 3272 1645 3428 \n","Q 1925 3584 2291 3584 \n","Q 2894 3584 3203 3211 \n","Q 3513 2838 3513 2113 \n","z\n","\" transform=\"scale(0.015625)\"/>\n","      </defs>\n","      <use xlink:href=\"#DejaVuSans-74\"/>\n","      <use xlink:href=\"#DejaVuSans-72\" x=\"39.208984\"/>\n","      <use xlink:href=\"#DejaVuSans-61\" x=\"80.322266\"/>\n","      <use xlink:href=\"#DejaVuSans-69\" x=\"141.601562\"/>\n","      <use xlink:href=\"#DejaVuSans-6e\" x=\"169.384766\"/>\n","     </g>\n","    </g>\n","    <g id=\"line2d_64\">\n","     <path d=\"M 60.378125 132.319531 \n","L 70.378125 132.319531 \n","L 80.378125 132.319531 \n","\" style=\"fill: none; stroke-dasharray: 5.55,2.4; stroke-dashoffset: 0; stroke: #bf00bf; stroke-width: 1.5\"/>\n","    </g>\n","    <g id=\"text_14\">\n","     <!-- test -->\n","     <g transform=\"translate(88.378125 135.819531)scale(0.1 -0.1)\">\n","      <use xlink:href=\"#DejaVuSans-74\"/>\n","      <use xlink:href=\"#DejaVuSans-65\" x=\"39.208984\"/>\n","      <use xlink:href=\"#DejaVuSans-73\" x=\"100.732422\"/>\n","      <use xlink:href=\"#DejaVuSans-74\" x=\"152.832031\"/>\n","     </g>\n","    </g>\n","   </g>\n","  </g>\n"," </g>\n"," <defs>\n","  <clipPath id=\"p1d7488a525\">\n","   <rect x=\"51.378125\" y=\"10.999219\" width=\"195.3\" height=\"135.9\"/>\n","  </clipPath>\n"," </defs>\n","</svg>\n"],"text/plain":["<Figure size 252x180 with 1 Axes>"]},"metadata":{"needs_background":"light"},"output_type":"display_data"}],"source":["# 从多项式特征中选择前2个维度，即1和x\n","train(poly_features[:n_train, :2], poly_features[n_train:, :2],\n","      labels[:n_train], labels[n_train:])"],"id":"f2532aad"},{"cell_type":"markdown","metadata":{"origin_pos":20,"id":"6aeced79"},"source":["### [**高阶多项式函数拟合(过拟合)**]\n","\n","现在，让我们尝试使用一个阶数过高的多项式来训练模型。\n","在这种情况下，没有足够的数据用于学到高阶系数应该具有接近于零的值。\n","因此，这个过于复杂的模型会轻易受到训练数据中噪声的影响。\n","虽然训练损失可以有效地降低，但测试损失仍然很高。\n","结果表明，复杂模型对数据造成了过拟合。\n"],"id":"6aeced79"},{"cell_type":"code","execution_count":null,"metadata":{"execution":{"iopub.execute_input":"2022-07-31T02:21:05.654356Z","iopub.status.busy":"2022-07-31T02:21:05.653854Z","iopub.status.idle":"2022-07-31T02:21:53.674609Z","shell.execute_reply":"2022-07-31T02:21:53.673951Z"},"origin_pos":21,"tab":["pytorch"],"id":"ac04a8f2","outputId":"f8d1a794-949d-4a40-b5da-7c230ed3f92f"},"outputs":[{"name":"stdout","output_type":"stream","text":["weight: [[ 4.9602804   1.2281249  -3.1678302   5.282293   -0.78401726  1.3070852\n","   0.05391492  0.08786786 -0.16313833 -0.02856293 -0.17763665  0.05075314\n","   0.14930737  0.1516932   0.11134355  0.18925151 -0.14107536  0.04467897\n","  -0.01062597 -0.09159212]]\n"]},{"data":{"image/svg+xml":["<?xml version=\"1.0\" encoding=\"utf-8\" standalone=\"no\"?>\n","<!DOCTYPE svg PUBLIC \"-//W3C//DTD SVG 1.1//EN\"\n","  \"http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd\">\n","<svg xmlns:xlink=\"http://www.w3.org/1999/xlink\" width=\"266.603125pt\" height=\"184.455469pt\" viewBox=\"0 0 266.603125 184.455469\" xmlns=\"http://www.w3.org/2000/svg\" version=\"1.1\">\n"," <metadata>\n","  <rdf:RDF xmlns:dc=\"http://purl.org/dc/elements/1.1/\" xmlns:cc=\"http://creativecommons.org/ns#\" xmlns:rdf=\"http://www.w3.org/1999/02/22-rdf-syntax-ns#\">\n","   <cc:Work>\n","    <dc:type rdf:resource=\"http://purl.org/dc/dcmitype/StillImage\"/>\n","    <dc:date>2022-07-31T02:21:53.523967</dc:date>\n","    <dc:format>image/svg+xml</dc:format>\n","    <dc:creator>\n","     <cc:Agent>\n","      <dc:title>Matplotlib v3.5.1, https://matplotlib.org/</dc:title>\n","     </cc:Agent>\n","    </dc:creator>\n","   </cc:Work>\n","  </rdf:RDF>\n"," </metadata>\n"," <defs>\n","  <style type=\"text/css\">*{stroke-linejoin: round; stroke-linecap: butt}</style>\n"," </defs>\n"," <g id=\"figure_1\">\n","  <g id=\"patch_1\">\n","   <path d=\"M 0 184.455469 \n","L 266.603125 184.455469 \n","L 266.603125 0 \n","L 0 0 \n","L 0 184.455469 \n","z\n","\" style=\"fill: none\"/>\n","  </g>\n","  <g id=\"axes_1\">\n","   <g id=\"patch_2\">\n","    <path d=\"M 51.378125 146.899219 \n","L 246.678125 146.899219 \n","L 246.678125 10.999219 \n","L 51.378125 10.999219 \n","z\n","\" style=\"fill: #ffffff\"/>\n","   </g>\n","   <g id=\"matplotlib.axis_1\">\n","    <g id=\"xtick_1\">\n","     <g id=\"line2d_1\">\n","      <path d=\"M 83.819553 146.899219 \n","L 83.819553 10.999219 \n","\" clip-path=\"url(#p44b2c4c390)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n","     </g>\n","     <g id=\"line2d_2\">\n","      <defs>\n","       <path id=\"mcaa03d4433\" d=\"M 0 0 \n","L 0 3.5 \n","\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n","      </defs>\n","      <g>\n","       <use xlink:href=\"#mcaa03d4433\" x=\"83.819553\" y=\"146.899219\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n","      </g>\n","     </g>\n","     <g id=\"text_1\">\n","      <!-- 250 -->\n","      <g transform=\"translate(74.275803 161.497656)scale(0.1 -0.1)\">\n","       <defs>\n","        <path id=\"DejaVuSans-32\" d=\"M 1228 531 \n","L 3431 531 \n","L 3431 0 \n","L 469 0 \n","L 469 531 \n","Q 828 903 1448 1529 \n","Q 2069 2156 2228 2338 \n","Q 2531 2678 2651 2914 \n","Q 2772 3150 2772 3378 \n","Q 2772 3750 2511 3984 \n","Q 2250 4219 1831 4219 \n","Q 1534 4219 1204 4116 \n","Q 875 4013 500 3803 \n","L 500 4441 \n","Q 881 4594 1212 4672 \n","Q 1544 4750 1819 4750 \n","Q 2544 4750 2975 4387 \n","Q 3406 4025 3406 3419 \n","Q 3406 3131 3298 2873 \n","Q 3191 2616 2906 2266 \n","Q 2828 2175 2409 1742 \n","Q 1991 1309 1228 531 \n","z\n","\" transform=\"scale(0.015625)\"/>\n","        <path id=\"DejaVuSans-35\" d=\"M 691 4666 \n","L 3169 4666 \n","L 3169 4134 \n","L 1269 4134 \n","L 1269 2991 \n","Q 1406 3038 1543 3061 \n","Q 1681 3084 1819 3084 \n","Q 2600 3084 3056 2656 \n","Q 3513 2228 3513 1497 \n","Q 3513 744 3044 326 \n","Q 2575 -91 1722 -91 \n","Q 1428 -91 1123 -41 \n","Q 819 9 494 109 \n","L 494 744 \n","Q 775 591 1075 516 \n","Q 1375 441 1709 441 \n","Q 2250 441 2565 725 \n","Q 2881 1009 2881 1497 \n","Q 2881 1984 2565 2268 \n","Q 2250 2553 1709 2553 \n","Q 1456 2553 1204 2497 \n","Q 953 2441 691 2322 \n","L 691 4666 \n","z\n","\" transform=\"scale(0.015625)\"/>\n","        <path id=\"DejaVuSans-30\" d=\"M 2034 4250 \n","Q 1547 4250 1301 3770 \n","Q 1056 3291 1056 2328 \n","Q 1056 1369 1301 889 \n","Q 1547 409 2034 409 \n","Q 2525 409 2770 889 \n","Q 3016 1369 3016 2328 \n","Q 3016 3291 2770 3770 \n","Q 2525 4250 2034 4250 \n","z\n","M 2034 4750 \n","Q 2819 4750 3233 4129 \n","Q 3647 3509 3647 2328 \n","Q 3647 1150 3233 529 \n","Q 2819 -91 2034 -91 \n","Q 1250 -91 836 529 \n","Q 422 1150 422 2328 \n","Q 422 3509 836 4129 \n","Q 1250 4750 2034 4750 \n","z\n","\" transform=\"scale(0.015625)\"/>\n","       </defs>\n","       <use xlink:href=\"#DejaVuSans-32\"/>\n","       <use xlink:href=\"#DejaVuSans-35\" x=\"63.623047\"/>\n","       <use xlink:href=\"#DejaVuSans-30\" x=\"127.246094\"/>\n","      </g>\n","     </g>\n","    </g>\n","    <g id=\"xtick_2\">\n","     <g id=\"line2d_3\">\n","      <path d=\"M 116.391267 146.899219 \n","L 116.391267 10.999219 \n","\" clip-path=\"url(#p44b2c4c390)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n","     </g>\n","     <g id=\"line2d_4\">\n","      <g>\n","       <use xlink:href=\"#mcaa03d4433\" x=\"116.391267\" y=\"146.899219\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n","      </g>\n","     </g>\n","     <g id=\"text_2\">\n","      <!-- 500 -->\n","      <g transform=\"translate(106.847517 161.497656)scale(0.1 -0.1)\">\n","       <use xlink:href=\"#DejaVuSans-35\"/>\n","       <use xlink:href=\"#DejaVuSans-30\" x=\"63.623047\"/>\n","       <use xlink:href=\"#DejaVuSans-30\" x=\"127.246094\"/>\n","      </g>\n","     </g>\n","    </g>\n","    <g id=\"xtick_3\">\n","     <g id=\"line2d_5\">\n","      <path d=\"M 148.962982 146.899219 \n","L 148.962982 10.999219 \n","\" clip-path=\"url(#p44b2c4c390)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n","     </g>\n","     <g id=\"line2d_6\">\n","      <g>\n","       <use xlink:href=\"#mcaa03d4433\" x=\"148.962982\" y=\"146.899219\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n","      </g>\n","     </g>\n","     <g id=\"text_3\">\n","      <!-- 750 -->\n","      <g transform=\"translate(139.419232 161.497656)scale(0.1 -0.1)\">\n","       <defs>\n","        <path id=\"DejaVuSans-37\" d=\"M 525 4666 \n","L 3525 4666 \n","L 3525 4397 \n","L 1831 0 \n","L 1172 0 \n","L 2766 4134 \n","L 525 4134 \n","L 525 4666 \n","z\n","\" transform=\"scale(0.015625)\"/>\n","       </defs>\n","       <use xlink:href=\"#DejaVuSans-37\"/>\n","       <use xlink:href=\"#DejaVuSans-35\" x=\"63.623047\"/>\n","       <use xlink:href=\"#DejaVuSans-30\" x=\"127.246094\"/>\n","      </g>\n","     </g>\n","    </g>\n","    <g id=\"xtick_4\">\n","     <g id=\"line2d_7\">\n","      <path d=\"M 181.534696 146.899219 \n","L 181.534696 10.999219 \n","\" clip-path=\"url(#p44b2c4c390)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n","     </g>\n","     <g id=\"line2d_8\">\n","      <g>\n","       <use xlink:href=\"#mcaa03d4433\" x=\"181.534696\" y=\"146.899219\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n","      </g>\n","     </g>\n","     <g id=\"text_4\">\n","      <!-- 1000 -->\n","      <g transform=\"translate(168.809696 161.497656)scale(0.1 -0.1)\">\n","       <defs>\n","        <path id=\"DejaVuSans-31\" d=\"M 794 531 \n","L 1825 531 \n","L 1825 4091 \n","L 703 3866 \n","L 703 4441 \n","L 1819 4666 \n","L 2450 4666 \n","L 2450 531 \n","L 3481 531 \n","L 3481 0 \n","L 794 0 \n","L 794 531 \n","z\n","\" transform=\"scale(0.015625)\"/>\n","       </defs>\n","       <use xlink:href=\"#DejaVuSans-31\"/>\n","       <use xlink:href=\"#DejaVuSans-30\" x=\"63.623047\"/>\n","       <use xlink:href=\"#DejaVuSans-30\" x=\"127.246094\"/>\n","       <use xlink:href=\"#DejaVuSans-30\" x=\"190.869141\"/>\n","      </g>\n","     </g>\n","    </g>\n","    <g id=\"xtick_5\">\n","     <g id=\"line2d_9\">\n","      <path d=\"M 214.106411 146.899219 \n","L 214.106411 10.999219 \n","\" clip-path=\"url(#p44b2c4c390)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n","     </g>\n","     <g id=\"line2d_10\">\n","      <g>\n","       <use xlink:href=\"#mcaa03d4433\" x=\"214.106411\" y=\"146.899219\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n","      </g>\n","     </g>\n","     <g id=\"text_5\">\n","      <!-- 1250 -->\n","      <g transform=\"translate(201.381411 161.497656)scale(0.1 -0.1)\">\n","       <use xlink:href=\"#DejaVuSans-31\"/>\n","       <use xlink:href=\"#DejaVuSans-32\" x=\"63.623047\"/>\n","       <use xlink:href=\"#DejaVuSans-35\" x=\"127.246094\"/>\n","       <use xlink:href=\"#DejaVuSans-30\" x=\"190.869141\"/>\n","      </g>\n","     </g>\n","    </g>\n","    <g id=\"xtick_6\">\n","     <g id=\"line2d_11\">\n","      <path d=\"M 246.678125 146.899219 \n","L 246.678125 10.999219 \n","\" clip-path=\"url(#p44b2c4c390)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n","     </g>\n","     <g id=\"line2d_12\">\n","      <g>\n","       <use xlink:href=\"#mcaa03d4433\" x=\"246.678125\" y=\"146.899219\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n","      </g>\n","     </g>\n","     <g id=\"text_6\">\n","      <!-- 1500 -->\n","      <g transform=\"translate(233.953125 161.497656)scale(0.1 -0.1)\">\n","       <use xlink:href=\"#DejaVuSans-31\"/>\n","       <use xlink:href=\"#DejaVuSans-35\" x=\"63.623047\"/>\n","       <use xlink:href=\"#DejaVuSans-30\" x=\"127.246094\"/>\n","       <use xlink:href=\"#DejaVuSans-30\" x=\"190.869141\"/>\n","      </g>\n","     </g>\n","    </g>\n","    <g id=\"text_7\">\n","     <!-- epoch -->\n","     <g transform=\"translate(133.8 175.175781)scale(0.1 -0.1)\">\n","      <defs>\n","       <path id=\"DejaVuSans-65\" d=\"M 3597 1894 \n","L 3597 1613 \n","L 953 1613 \n","Q 991 1019 1311 708 \n","Q 1631 397 2203 397 \n","Q 2534 397 2845 478 \n","Q 3156 559 3463 722 \n","L 3463 178 \n","Q 3153 47 2828 -22 \n","Q 2503 -91 2169 -91 \n","Q 1331 -91 842 396 \n","Q 353 884 353 1716 \n","Q 353 2575 817 3079 \n","Q 1281 3584 2069 3584 \n","Q 2775 3584 3186 3129 \n","Q 3597 2675 3597 1894 \n","z\n","M 3022 2063 \n","Q 3016 2534 2758 2815 \n","Q 2500 3097 2075 3097 \n","Q 1594 3097 1305 2825 \n","Q 1016 2553 972 2059 \n","L 3022 2063 \n","z\n","\" transform=\"scale(0.015625)\"/>\n","       <path id=\"DejaVuSans-70\" d=\"M 1159 525 \n","L 1159 -1331 \n","L 581 -1331 \n","L 581 3500 \n","L 1159 3500 \n","L 1159 2969 \n","Q 1341 3281 1617 3432 \n","Q 1894 3584 2278 3584 \n","Q 2916 3584 3314 3078 \n","Q 3713 2572 3713 1747 \n","Q 3713 922 3314 415 \n","Q 2916 -91 2278 -91 \n","Q 1894 -91 1617 61 \n","Q 1341 213 1159 525 \n","z\n","M 3116 1747 \n","Q 3116 2381 2855 2742 \n","Q 2594 3103 2138 3103 \n","Q 1681 3103 1420 2742 \n","Q 1159 2381 1159 1747 \n","Q 1159 1113 1420 752 \n","Q 1681 391 2138 391 \n","Q 2594 391 2855 752 \n","Q 3116 1113 3116 1747 \n","z\n","\" transform=\"scale(0.015625)\"/>\n","       <path id=\"DejaVuSans-6f\" d=\"M 1959 3097 \n","Q 1497 3097 1228 2736 \n","Q 959 2375 959 1747 \n","Q 959 1119 1226 758 \n","Q 1494 397 1959 397 \n","Q 2419 397 2687 759 \n","Q 2956 1122 2956 1747 \n","Q 2956 2369 2687 2733 \n","Q 2419 3097 1959 3097 \n","z\n","M 1959 3584 \n","Q 2709 3584 3137 3096 \n","Q 3566 2609 3566 1747 \n","Q 3566 888 3137 398 \n","Q 2709 -91 1959 -91 \n","Q 1206 -91 779 398 \n","Q 353 888 353 1747 \n","Q 353 2609 779 3096 \n","Q 1206 3584 1959 3584 \n","z\n","\" transform=\"scale(0.015625)\"/>\n","       <path id=\"DejaVuSans-63\" d=\"M 3122 3366 \n","L 3122 2828 \n","Q 2878 2963 2633 3030 \n","Q 2388 3097 2138 3097 \n","Q 1578 3097 1268 2742 \n","Q 959 2388 959 1747 \n","Q 959 1106 1268 751 \n","Q 1578 397 2138 397 \n","Q 2388 397 2633 464 \n","Q 2878 531 3122 666 \n","L 3122 134 \n","Q 2881 22 2623 -34 \n","Q 2366 -91 2075 -91 \n","Q 1284 -91 818 406 \n","Q 353 903 353 1747 \n","Q 353 2603 823 3093 \n","Q 1294 3584 2113 3584 \n","Q 2378 3584 2631 3529 \n","Q 2884 3475 3122 3366 \n","z\n","\" transform=\"scale(0.015625)\"/>\n","       <path id=\"DejaVuSans-68\" d=\"M 3513 2113 \n","L 3513 0 \n","L 2938 0 \n","L 2938 2094 \n","Q 2938 2591 2744 2837 \n","Q 2550 3084 2163 3084 \n","Q 1697 3084 1428 2787 \n","Q 1159 2491 1159 1978 \n","L 1159 0 \n","L 581 0 \n","L 581 4863 \n","L 1159 4863 \n","L 1159 2956 \n","Q 1366 3272 1645 3428 \n","Q 1925 3584 2291 3584 \n","Q 2894 3584 3203 3211 \n","Q 3513 2838 3513 2113 \n","z\n","\" transform=\"scale(0.015625)\"/>\n","      </defs>\n","      <use xlink:href=\"#DejaVuSans-65\"/>\n","      <use xlink:href=\"#DejaVuSans-70\" x=\"61.523438\"/>\n","      <use xlink:href=\"#DejaVuSans-6f\" x=\"125\"/>\n","      <use xlink:href=\"#DejaVuSans-63\" x=\"186.181641\"/>\n","      <use xlink:href=\"#DejaVuSans-68\" x=\"241.162109\"/>\n","     </g>\n","    </g>\n","   </g>\n","   <g id=\"matplotlib.axis_2\">\n","    <g id=\"ytick_1\">\n","     <g id=\"line2d_13\">\n","      <path d=\"M 51.378125 146.899219 \n","L 246.678125 146.899219 \n","\" clip-path=\"url(#p44b2c4c390)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n","     </g>\n","     <g id=\"line2d_14\">\n","      <defs>\n","       <path id=\"m889814a038\" d=\"M 0 0 \n","L -3.5 0 \n","\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n","      </defs>\n","      <g>\n","       <use xlink:href=\"#m889814a038\" x=\"51.378125\" y=\"146.899219\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n","      </g>\n","     </g>\n","     <g id=\"text_8\">\n","      <!-- $\\mathdefault{10^{-3}}$ -->\n","      <g transform=\"translate(20.878125 150.698438)scale(0.1 -0.1)\">\n","       <defs>\n","        <path id=\"DejaVuSans-2212\" d=\"M 678 2272 \n","L 4684 2272 \n","L 4684 1741 \n","L 678 1741 \n","L 678 2272 \n","z\n","\" transform=\"scale(0.015625)\"/>\n","        <path id=\"DejaVuSans-33\" d=\"M 2597 2516 \n","Q 3050 2419 3304 2112 \n","Q 3559 1806 3559 1356 \n","Q 3559 666 3084 287 \n","Q 2609 -91 1734 -91 \n","Q 1441 -91 1130 -33 \n","Q 819 25 488 141 \n","L 488 750 \n","Q 750 597 1062 519 \n","Q 1375 441 1716 441 \n","Q 2309 441 2620 675 \n","Q 2931 909 2931 1356 \n","Q 2931 1769 2642 2001 \n","Q 2353 2234 1838 2234 \n","L 1294 2234 \n","L 1294 2753 \n","L 1863 2753 \n","Q 2328 2753 2575 2939 \n","Q 2822 3125 2822 3475 \n","Q 2822 3834 2567 4026 \n","Q 2313 4219 1838 4219 \n","Q 1578 4219 1281 4162 \n","Q 984 4106 628 3988 \n","L 628 4550 \n","Q 988 4650 1302 4700 \n","Q 1616 4750 1894 4750 \n","Q 2613 4750 3031 4423 \n","Q 3450 4097 3450 3541 \n","Q 3450 3153 3228 2886 \n","Q 3006 2619 2597 2516 \n","z\n","\" transform=\"scale(0.015625)\"/>\n","       </defs>\n","       <use xlink:href=\"#DejaVuSans-31\" transform=\"translate(0 0.765625)\"/>\n","       <use xlink:href=\"#DejaVuSans-30\" transform=\"translate(63.623047 0.765625)\"/>\n","       <use xlink:href=\"#DejaVuSans-2212\" transform=\"translate(128.203125 39.046875)scale(0.7)\"/>\n","       <use xlink:href=\"#DejaVuSans-33\" transform=\"translate(186.855469 39.046875)scale(0.7)\"/>\n","      </g>\n","     </g>\n","    </g>\n","    <g id=\"ytick_2\">\n","     <g id=\"line2d_15\">\n","      <path d=\"M 51.378125 119.719219 \n","L 246.678125 119.719219 \n","\" clip-path=\"url(#p44b2c4c390)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n","     </g>\n","     <g id=\"line2d_16\">\n","      <g>\n","       <use xlink:href=\"#m889814a038\" x=\"51.378125\" y=\"119.719219\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n","      </g>\n","     </g>\n","     <g id=\"text_9\">\n","      <!-- $\\mathdefault{10^{-2}}$ -->\n","      <g transform=\"translate(20.878125 123.518438)scale(0.1 -0.1)\">\n","       <use xlink:href=\"#DejaVuSans-31\" transform=\"translate(0 0.765625)\"/>\n","       <use xlink:href=\"#DejaVuSans-30\" transform=\"translate(63.623047 0.765625)\"/>\n","       <use xlink:href=\"#DejaVuSans-2212\" transform=\"translate(128.203125 39.046875)scale(0.7)\"/>\n","       <use xlink:href=\"#DejaVuSans-32\" transform=\"translate(186.855469 39.046875)scale(0.7)\"/>\n","      </g>\n","     </g>\n","    </g>\n","    <g id=\"ytick_3\">\n","     <g id=\"line2d_17\">\n","      <path d=\"M 51.378125 92.539219 \n","L 246.678125 92.539219 \n","\" clip-path=\"url(#p44b2c4c390)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n","     </g>\n","     <g id=\"line2d_18\">\n","      <g>\n","       <use xlink:href=\"#m889814a038\" x=\"51.378125\" y=\"92.539219\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n","      </g>\n","     </g>\n","     <g id=\"text_10\">\n","      <!-- $\\mathdefault{10^{-1}}$ -->\n","      <g transform=\"translate(20.878125 96.338437)scale(0.1 -0.1)\">\n","       <use xlink:href=\"#DejaVuSans-31\" transform=\"translate(0 0.684375)\"/>\n","       <use xlink:href=\"#DejaVuSans-30\" transform=\"translate(63.623047 0.684375)\"/>\n","       <use xlink:href=\"#DejaVuSans-2212\" transform=\"translate(128.203125 38.965625)scale(0.7)\"/>\n","       <use xlink:href=\"#DejaVuSans-31\" transform=\"translate(186.855469 38.965625)scale(0.7)\"/>\n","      </g>\n","     </g>\n","    </g>\n","    <g id=\"ytick_4\">\n","     <g id=\"line2d_19\">\n","      <path d=\"M 51.378125 65.359219 \n","L 246.678125 65.359219 \n","\" clip-path=\"url(#p44b2c4c390)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n","     </g>\n","     <g id=\"line2d_20\">\n","      <g>\n","       <use xlink:href=\"#m889814a038\" x=\"51.378125\" y=\"65.359219\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n","      </g>\n","     </g>\n","     <g id=\"text_11\">\n","      <!-- $\\mathdefault{10^{0}}$ -->\n","      <g transform=\"translate(26.778125 69.158437)scale(0.1 -0.1)\">\n","       <use xlink:href=\"#DejaVuSans-31\" transform=\"translate(0 0.765625)\"/>\n","       <use xlink:href=\"#DejaVuSans-30\" transform=\"translate(63.623047 0.765625)\"/>\n","       <use xlink:href=\"#DejaVuSans-30\" transform=\"translate(128.203125 39.046875)scale(0.7)\"/>\n","      </g>\n","     </g>\n","    </g>\n","    <g id=\"ytick_5\">\n","     <g id=\"line2d_21\">\n","      <path d=\"M 51.378125 38.179219 \n","L 246.678125 38.179219 \n","\" clip-path=\"url(#p44b2c4c390)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n","     </g>\n","     <g id=\"line2d_22\">\n","      <g>\n","       <use xlink:href=\"#m889814a038\" x=\"51.378125\" y=\"38.179219\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n","      </g>\n","     </g>\n","     <g id=\"text_12\">\n","      <!-- $\\mathdefault{10^{1}}$ -->\n","      <g transform=\"translate(26.778125 41.978437)scale(0.1 -0.1)\">\n","       <use xlink:href=\"#DejaVuSans-31\" transform=\"translate(0 0.684375)\"/>\n","       <use xlink:href=\"#DejaVuSans-30\" transform=\"translate(63.623047 0.684375)\"/>\n","       <use xlink:href=\"#DejaVuSans-31\" transform=\"translate(128.203125 38.965625)scale(0.7)\"/>\n","      </g>\n","     </g>\n","    </g>\n","    <g id=\"ytick_6\">\n","     <g id=\"line2d_23\">\n","      <path d=\"M 51.378125 10.999219 \n","L 246.678125 10.999219 \n","\" clip-path=\"url(#p44b2c4c390)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n","     </g>\n","     <g id=\"line2d_24\">\n","      <g>\n","       <use xlink:href=\"#m889814a038\" x=\"51.378125\" y=\"10.999219\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n","      </g>\n","     </g>\n","     <g id=\"text_13\">\n","      <!-- $\\mathdefault{10^{2}}$ -->\n","      <g transform=\"translate(26.778125 14.798437)scale(0.1 -0.1)\">\n","       <use xlink:href=\"#DejaVuSans-31\" transform=\"translate(0 0.765625)\"/>\n","       <use xlink:href=\"#DejaVuSans-30\" transform=\"translate(63.623047 0.765625)\"/>\n","       <use xlink:href=\"#DejaVuSans-32\" transform=\"translate(128.203125 39.046875)scale(0.7)\"/>\n","      </g>\n","     </g>\n","    </g>\n","    <g id=\"ytick_7\">\n","     <g id=\"line2d_25\">\n","      <defs>\n","       <path id=\"m962d829aa8\" d=\"M 0 0 \n","L -2 0 \n","\" style=\"stroke: #000000; stroke-width: 0.6\"/>\n","      </defs>\n","      <g>\n","       <use xlink:href=\"#m962d829aa8\" x=\"51.378125\" y=\"138.717223\" style=\"stroke: #000000; stroke-width: 0.6\"/>\n","      </g>\n","     </g>\n","    </g>\n","    <g id=\"ytick_8\">\n","     <g id=\"line2d_26\">\n","      <g>\n","       <use xlink:href=\"#m962d829aa8\" x=\"51.378125\" y=\"133.931063\" style=\"stroke: #000000; stroke-width: 0.6\"/>\n","      </g>\n","     </g>\n","    </g>\n","    <g id=\"ytick_9\">\n","     <g id=\"line2d_27\">\n","      <g>\n","       <use xlink:href=\"#m962d829aa8\" x=\"51.378125\" y=\"130.535228\" style=\"stroke: #000000; stroke-width: 0.6\"/>\n","      </g>\n","     </g>\n","    </g>\n","    <g id=\"ytick_10\">\n","     <g id=\"line2d_28\">\n","      <g>\n","       <use xlink:href=\"#m962d829aa8\" x=\"51.378125\" y=\"127.901214\" style=\"stroke: #000000; stroke-width: 0.6\"/>\n","      </g>\n","     </g>\n","    </g>\n","    <g id=\"ytick_11\">\n","     <g id=\"line2d_29\">\n","      <g>\n","       <use xlink:href=\"#m962d829aa8\" x=\"51.378125\" y=\"125.749068\" style=\"stroke: #000000; stroke-width: 0.6\"/>\n","      </g>\n","     </g>\n","    </g>\n","    <g id=\"ytick_12\">\n","     <g id=\"line2d_30\">\n","      <g>\n","       <use xlink:href=\"#m962d829aa8\" x=\"51.378125\" y=\"123.929454\" style=\"stroke: #000000; stroke-width: 0.6\"/>\n","      </g>\n","     </g>\n","    </g>\n","    <g id=\"ytick_13\">\n","     <g id=\"line2d_31\">\n","      <g>\n","       <use xlink:href=\"#m962d829aa8\" x=\"51.378125\" y=\"122.353233\" style=\"stroke: #000000; stroke-width: 0.6\"/>\n","      </g>\n","     </g>\n","    </g>\n","    <g id=\"ytick_14\">\n","     <g id=\"line2d_32\">\n","      <g>\n","       <use xlink:href=\"#m962d829aa8\" x=\"51.378125\" y=\"120.962907\" style=\"stroke: #000000; stroke-width: 0.6\"/>\n","      </g>\n","     </g>\n","    </g>\n","    <g id=\"ytick_15\">\n","     <g id=\"line2d_33\">\n","      <g>\n","       <use xlink:href=\"#m962d829aa8\" x=\"51.378125\" y=\"111.537223\" style=\"stroke: #000000; stroke-width: 0.6\"/>\n","      </g>\n","     </g>\n","    </g>\n","    <g id=\"ytick_16\">\n","     <g id=\"line2d_34\">\n","      <g>\n","       <use xlink:href=\"#m962d829aa8\" x=\"51.378125\" y=\"106.751063\" style=\"stroke: #000000; stroke-width: 0.6\"/>\n","      </g>\n","     </g>\n","    </g>\n","    <g id=\"ytick_17\">\n","     <g id=\"line2d_35\">\n","      <g>\n","       <use xlink:href=\"#m962d829aa8\" x=\"51.378125\" y=\"103.355228\" style=\"stroke: #000000; stroke-width: 0.6\"/>\n","      </g>\n","     </g>\n","    </g>\n","    <g id=\"ytick_18\">\n","     <g id=\"line2d_36\">\n","      <g>\n","       <use xlink:href=\"#m962d829aa8\" x=\"51.378125\" y=\"100.721214\" style=\"stroke: #000000; stroke-width: 0.6\"/>\n","      </g>\n","     </g>\n","    </g>\n","    <g id=\"ytick_19\">\n","     <g id=\"line2d_37\">\n","      <g>\n","       <use xlink:href=\"#m962d829aa8\" x=\"51.378125\" y=\"98.569068\" style=\"stroke: #000000; stroke-width: 0.6\"/>\n","      </g>\n","     </g>\n","    </g>\n","    <g id=\"ytick_20\">\n","     <g id=\"line2d_38\">\n","      <g>\n","       <use xlink:href=\"#m962d829aa8\" x=\"51.378125\" y=\"96.749454\" style=\"stroke: #000000; stroke-width: 0.6\"/>\n","      </g>\n","     </g>\n","    </g>\n","    <g id=\"ytick_21\">\n","     <g id=\"line2d_39\">\n","      <g>\n","       <use xlink:href=\"#m962d829aa8\" x=\"51.378125\" y=\"95.173233\" style=\"stroke: #000000; stroke-width: 0.6\"/>\n","      </g>\n","     </g>\n","    </g>\n","    <g id=\"ytick_22\">\n","     <g id=\"line2d_40\">\n","      <g>\n","       <use xlink:href=\"#m962d829aa8\" x=\"51.378125\" y=\"93.782907\" style=\"stroke: #000000; stroke-width: 0.6\"/>\n","      </g>\n","     </g>\n","    </g>\n","    <g id=\"ytick_23\">\n","     <g id=\"line2d_41\">\n","      <g>\n","       <use xlink:href=\"#m962d829aa8\" x=\"51.378125\" y=\"84.357223\" style=\"stroke: #000000; stroke-width: 0.6\"/>\n","      </g>\n","     </g>\n","    </g>\n","    <g id=\"ytick_24\">\n","     <g id=\"line2d_42\">\n","      <g>\n","       <use xlink:href=\"#m962d829aa8\" x=\"51.378125\" y=\"79.571063\" style=\"stroke: #000000; stroke-width: 0.6\"/>\n","      </g>\n","     </g>\n","    </g>\n","    <g id=\"ytick_25\">\n","     <g id=\"line2d_43\">\n","      <g>\n","       <use xlink:href=\"#m962d829aa8\" x=\"51.378125\" y=\"76.175228\" style=\"stroke: #000000; stroke-width: 0.6\"/>\n","      </g>\n","     </g>\n","    </g>\n","    <g id=\"ytick_26\">\n","     <g id=\"line2d_44\">\n","      <g>\n","       <use xlink:href=\"#m962d829aa8\" x=\"51.378125\" y=\"73.541214\" style=\"stroke: #000000; stroke-width: 0.6\"/>\n","      </g>\n","     </g>\n","    </g>\n","    <g id=\"ytick_27\">\n","     <g id=\"line2d_45\">\n","      <g>\n","       <use xlink:href=\"#m962d829aa8\" x=\"51.378125\" y=\"71.389068\" style=\"stroke: #000000; stroke-width: 0.6\"/>\n","      </g>\n","     </g>\n","    </g>\n","    <g id=\"ytick_28\">\n","     <g id=\"line2d_46\">\n","      <g>\n","       <use xlink:href=\"#m962d829aa8\" x=\"51.378125\" y=\"69.569454\" style=\"stroke: #000000; stroke-width: 0.6\"/>\n","      </g>\n","     </g>\n","    </g>\n","    <g id=\"ytick_29\">\n","     <g id=\"line2d_47\">\n","      <g>\n","       <use xlink:href=\"#m962d829aa8\" x=\"51.378125\" y=\"67.993233\" style=\"stroke: #000000; stroke-width: 0.6\"/>\n","      </g>\n","     </g>\n","    </g>\n","    <g id=\"ytick_30\">\n","     <g id=\"line2d_48\">\n","      <g>\n","       <use xlink:href=\"#m962d829aa8\" x=\"51.378125\" y=\"66.602907\" style=\"stroke: #000000; stroke-width: 0.6\"/>\n","      </g>\n","     </g>\n","    </g>\n","    <g id=\"ytick_31\">\n","     <g id=\"line2d_49\">\n","      <g>\n","       <use xlink:href=\"#m962d829aa8\" x=\"51.378125\" y=\"57.177223\" style=\"stroke: #000000; stroke-width: 0.6\"/>\n","      </g>\n","     </g>\n","    </g>\n","    <g id=\"ytick_32\">\n","     <g id=\"line2d_50\">\n","      <g>\n","       <use xlink:href=\"#m962d829aa8\" x=\"51.378125\" y=\"52.391063\" style=\"stroke: #000000; stroke-width: 0.6\"/>\n","      </g>\n","     </g>\n","    </g>\n","    <g id=\"ytick_33\">\n","     <g id=\"line2d_51\">\n","      <g>\n","       <use xlink:href=\"#m962d829aa8\" x=\"51.378125\" y=\"48.995228\" style=\"stroke: #000000; stroke-width: 0.6\"/>\n","      </g>\n","     </g>\n","    </g>\n","    <g id=\"ytick_34\">\n","     <g id=\"line2d_52\">\n","      <g>\n","       <use xlink:href=\"#m962d829aa8\" x=\"51.378125\" y=\"46.361214\" style=\"stroke: #000000; stroke-width: 0.6\"/>\n","      </g>\n","     </g>\n","    </g>\n","    <g id=\"ytick_35\">\n","     <g id=\"line2d_53\">\n","      <g>\n","       <use xlink:href=\"#m962d829aa8\" x=\"51.378125\" y=\"44.209068\" style=\"stroke: #000000; stroke-width: 0.6\"/>\n","      </g>\n","     </g>\n","    </g>\n","    <g id=\"ytick_36\">\n","     <g id=\"line2d_54\">\n","      <g>\n","       <use xlink:href=\"#m962d829aa8\" x=\"51.378125\" y=\"42.389454\" style=\"stroke: #000000; stroke-width: 0.6\"/>\n","      </g>\n","     </g>\n","    </g>\n","    <g id=\"ytick_37\">\n","     <g id=\"line2d_55\">\n","      <g>\n","       <use xlink:href=\"#m962d829aa8\" x=\"51.378125\" y=\"40.813233\" style=\"stroke: #000000; stroke-width: 0.6\"/>\n","      </g>\n","     </g>\n","    </g>\n","    <g id=\"ytick_38\">\n","     <g id=\"line2d_56\">\n","      <g>\n","       <use xlink:href=\"#m962d829aa8\" x=\"51.378125\" y=\"39.422907\" style=\"stroke: #000000; stroke-width: 0.6\"/>\n","      </g>\n","     </g>\n","    </g>\n","    <g id=\"ytick_39\">\n","     <g id=\"line2d_57\">\n","      <g>\n","       <use xlink:href=\"#m962d829aa8\" x=\"51.378125\" y=\"29.997223\" style=\"stroke: #000000; stroke-width: 0.6\"/>\n","      </g>\n","     </g>\n","    </g>\n","    <g id=\"ytick_40\">\n","     <g id=\"line2d_58\">\n","      <g>\n","       <use xlink:href=\"#m962d829aa8\" x=\"51.378125\" y=\"25.211063\" style=\"stroke: #000000; stroke-width: 0.6\"/>\n","      </g>\n","     </g>\n","    </g>\n","    <g id=\"ytick_41\">\n","     <g id=\"line2d_59\">\n","      <g>\n","       <use xlink:href=\"#m962d829aa8\" x=\"51.378125\" y=\"21.815228\" style=\"stroke: #000000; stroke-width: 0.6\"/>\n","      </g>\n","     </g>\n","    </g>\n","    <g id=\"ytick_42\">\n","     <g id=\"line2d_60\">\n","      <g>\n","       <use xlink:href=\"#m962d829aa8\" x=\"51.378125\" y=\"19.181214\" style=\"stroke: #000000; stroke-width: 0.6\"/>\n","      </g>\n","     </g>\n","    </g>\n","    <g id=\"ytick_43\">\n","     <g id=\"line2d_61\">\n","      <g>\n","       <use xlink:href=\"#m962d829aa8\" x=\"51.378125\" y=\"17.029068\" style=\"stroke: #000000; stroke-width: 0.6\"/>\n","      </g>\n","     </g>\n","    </g>\n","    <g id=\"ytick_44\">\n","     <g id=\"line2d_62\">\n","      <g>\n","       <use xlink:href=\"#m962d829aa8\" x=\"51.378125\" y=\"15.209454\" style=\"stroke: #000000; stroke-width: 0.6\"/>\n","      </g>\n","     </g>\n","    </g>\n","    <g id=\"ytick_45\">\n","     <g id=\"line2d_63\">\n","      <g>\n","       <use xlink:href=\"#m962d829aa8\" x=\"51.378125\" y=\"13.633233\" style=\"stroke: #000000; stroke-width: 0.6\"/>\n","      </g>\n","     </g>\n","    </g>\n","    <g id=\"ytick_46\">\n","     <g id=\"line2d_64\">\n","      <g>\n","       <use xlink:href=\"#m962d829aa8\" x=\"51.378125\" y=\"12.242907\" style=\"stroke: #000000; stroke-width: 0.6\"/>\n","      </g>\n","     </g>\n","    </g>\n","    <g id=\"text_14\">\n","     <!-- loss -->\n","     <g transform=\"translate(14.798437 88.607031)rotate(-90)scale(0.1 -0.1)\">\n","      <defs>\n","       <path id=\"DejaVuSans-6c\" d=\"M 603 4863 \n","L 1178 4863 \n","L 1178 0 \n","L 603 0 \n","L 603 4863 \n","z\n","\" transform=\"scale(0.015625)\"/>\n","       <path id=\"DejaVuSans-73\" d=\"M 2834 3397 \n","L 2834 2853 \n","Q 2591 2978 2328 3040 \n","Q 2066 3103 1784 3103 \n","Q 1356 3103 1142 2972 \n","Q 928 2841 928 2578 \n","Q 928 2378 1081 2264 \n","Q 1234 2150 1697 2047 \n","L 1894 2003 \n","Q 2506 1872 2764 1633 \n","Q 3022 1394 3022 966 \n","Q 3022 478 2636 193 \n","Q 2250 -91 1575 -91 \n","Q 1294 -91 989 -36 \n","Q 684 19 347 128 \n","L 347 722 \n","Q 666 556 975 473 \n","Q 1284 391 1588 391 \n","Q 1994 391 2212 530 \n","Q 2431 669 2431 922 \n","Q 2431 1156 2273 1281 \n","Q 2116 1406 1581 1522 \n","L 1381 1569 \n","Q 847 1681 609 1914 \n","Q 372 2147 372 2553 \n","Q 372 3047 722 3315 \n","Q 1072 3584 1716 3584 \n","Q 2034 3584 2315 3537 \n","Q 2597 3491 2834 3397 \n","z\n","\" transform=\"scale(0.015625)\"/>\n","      </defs>\n","      <use xlink:href=\"#DejaVuSans-6c\"/>\n","      <use xlink:href=\"#DejaVuSans-6f\" x=\"27.783203\"/>\n","      <use xlink:href=\"#DejaVuSans-73\" x=\"88.964844\"/>\n","      <use xlink:href=\"#DejaVuSans-73\" x=\"141.064453\"/>\n","     </g>\n","    </g>\n","   </g>\n","   <g id=\"line2d_65\">\n","    <path d=\"M 51.378125 29.637998 \n","L 53.853575 58.369047 \n","L 56.459312 73.059439 \n","L 59.06505 85.726097 \n","L 61.670787 95.599215 \n","L 64.276524 101.710827 \n","L 66.882261 104.818937 \n","L 69.487998 106.419485 \n","L 72.093735 107.420548 \n","L 74.699473 108.208348 \n","L 77.30521 108.886277 \n","L 79.910947 109.501426 \n","L 82.516684 110.06817 \n","L 85.122421 110.589726 \n","L 87.728158 111.071623 \n","L 90.333896 111.513223 \n","L 92.939633 111.92254 \n","L 95.54537 112.296111 \n","L 98.151107 112.636785 \n","L 100.756844 112.949363 \n","L 103.362581 113.228845 \n","L 105.968318 113.492858 \n","L 108.574056 113.725908 \n","L 111.179793 113.941197 \n","L 113.78553 114.132659 \n","L 116.391267 114.309857 \n","L 118.997004 114.470452 \n","L 121.602741 114.614788 \n","L 124.208479 114.747178 \n","L 126.814216 114.866099 \n","L 129.419953 114.974718 \n","L 132.02569 115.073447 \n","L 134.631427 115.164132 \n","L 137.237164 115.246269 \n","L 139.842902 115.322315 \n","L 142.448639 115.390904 \n","L 145.054376 115.455425 \n","L 147.660113 115.51426 \n","L 150.26585 115.568109 \n","L 152.871587 115.619212 \n","L 155.477324 115.665102 \n","L 158.083062 115.709345 \n","L 160.688799 115.751093 \n","L 163.294536 115.789041 \n","L 165.900273 115.82682 \n","L 168.50601 115.859352 \n","L 171.111747 115.893519 \n","L 173.717485 115.925214 \n","L 176.323222 115.954745 \n","L 178.928959 115.982298 \n","L 181.534696 116.007666 \n","L 184.140433 116.037552 \n","L 186.74617 116.063005 \n","L 189.351908 116.087017 \n","L 191.957645 116.112313 \n","L 194.563382 116.135538 \n","L 197.169119 116.158051 \n","L 199.774856 116.179531 \n","L 202.380593 116.203119 \n","L 204.98633 116.224585 \n","L 207.592068 116.245339 \n","L 210.197805 116.266193 \n","L 212.803542 116.286694 \n","L 215.409279 116.307063 \n","L 218.015016 116.327207 \n","L 220.620753 116.346802 \n","L 223.226491 116.366401 \n","L 225.832228 116.385109 \n","L 228.437965 116.404667 \n","L 231.043702 116.423687 \n","L 233.649439 116.44218 \n","L 236.255176 116.460845 \n","L 238.860914 116.478885 \n","L 241.466651 116.497169 \n","L 244.072388 116.516014 \n","L 246.678125 116.534084 \n","\" clip-path=\"url(#p44b2c4c390)\" style=\"fill: none; stroke: #1f77b4; stroke-width: 1.5; stroke-linecap: square\"/>\n","   </g>\n","   <g id=\"line2d_66\">\n","    <path d=\"M 51.378125 25.55981 \n","L 53.853575 51.570065 \n","L 56.459312 67.527195 \n","L 59.06505 81.368864 \n","L 61.670787 92.939891 \n","L 64.276524 100.662826 \n","L 66.882261 104.249577 \n","L 69.487998 105.473704 \n","L 72.093735 105.659988 \n","L 74.699473 105.88818 \n","L 77.30521 105.863322 \n","L 79.910947 105.825523 \n","L 82.516684 105.766347 \n","L 85.122421 105.57967 \n","L 87.728158 105.420894 \n","L 90.333896 105.175484 \n","L 92.939633 104.960578 \n","L 95.54537 104.722882 \n","L 98.151107 104.456916 \n","L 100.756844 104.254043 \n","L 103.362581 103.888639 \n","L 105.968318 103.685143 \n","L 108.574056 103.39969 \n","L 111.179793 103.181439 \n","L 113.78553 102.913522 \n","L 116.391267 102.715528 \n","L 118.997004 102.511088 \n","L 121.602741 102.322264 \n","L 124.208479 102.139058 \n","L 126.814216 101.947124 \n","L 129.419953 101.80578 \n","L 132.02569 101.618375 \n","L 134.631427 101.519741 \n","L 137.237164 101.406229 \n","L 139.842902 101.241705 \n","L 142.448639 101.126418 \n","L 145.054376 101.027095 \n","L 147.660113 100.926129 \n","L 150.26585 100.856237 \n","L 152.871587 100.805441 \n","L 155.477324 100.718958 \n","L 158.083062 100.658096 \n","L 160.688799 100.617946 \n","L 163.294536 100.566643 \n","L 165.900273 100.534903 \n","L 168.50601 100.448242 \n","L 171.111747 100.430871 \n","L 173.717485 100.436873 \n","L 176.323222 100.404423 \n","L 178.928959 100.423278 \n","L 181.534696 100.399872 \n","L 184.140433 100.420482 \n","L 186.74617 100.397723 \n","L 189.351908 100.368423 \n","L 191.957645 100.417739 \n","L 194.563382 100.409543 \n","L 197.169119 100.422467 \n","L 199.774856 100.468073 \n","L 202.380593 100.48396 \n","L 204.98633 100.511418 \n","L 207.592068 100.554614 \n","L 210.197805 100.576821 \n","L 212.803542 100.551047 \n","L 215.409279 100.613368 \n","L 218.015016 100.649018 \n","L 220.620753 100.659139 \n","L 223.226491 100.703493 \n","L 225.832228 100.742558 \n","L 228.437965 100.762644 \n","L 231.043702 100.818065 \n","L 233.649439 100.866648 \n","L 236.255176 100.931644 \n","L 238.860914 100.935378 \n","L 241.466651 101.037541 \n","L 244.072388 101.049912 \n","L 246.678125 101.105785 \n","\" clip-path=\"url(#p44b2c4c390)\" style=\"fill: none; stroke-dasharray: 5.55,2.4; stroke-dashoffset: 0; stroke: #bf00bf; stroke-width: 1.5\"/>\n","   </g>\n","   <g id=\"patch_3\">\n","    <path d=\"M 51.378125 146.899219 \n","L 51.378125 10.999219 \n","\" style=\"fill: none; stroke: #000000; stroke-width: 0.8; stroke-linejoin: miter; stroke-linecap: square\"/>\n","   </g>\n","   <g id=\"patch_4\">\n","    <path d=\"M 246.678125 146.899219 \n","L 246.678125 10.999219 \n","\" style=\"fill: none; stroke: #000000; stroke-width: 0.8; stroke-linejoin: miter; stroke-linecap: square\"/>\n","   </g>\n","   <g id=\"patch_5\">\n","    <path d=\"M 51.378125 146.899219 \n","L 246.678125 146.899219 \n","\" style=\"fill: none; stroke: #000000; stroke-width: 0.8; stroke-linejoin: miter; stroke-linecap: square\"/>\n","   </g>\n","   <g id=\"patch_6\">\n","    <path d=\"M 51.378125 10.999219 \n","L 246.678125 10.999219 \n","\" style=\"fill: none; stroke: #000000; stroke-width: 0.8; stroke-linejoin: miter; stroke-linecap: square\"/>\n","   </g>\n","   <g id=\"legend_1\">\n","    <g id=\"patch_7\">\n","     <path d=\"M 184.403125 48.355469 \n","L 239.678125 48.355469 \n","Q 241.678125 48.355469 241.678125 46.355469 \n","L 241.678125 17.999219 \n","Q 241.678125 15.999219 239.678125 15.999219 \n","L 184.403125 15.999219 \n","Q 182.403125 15.999219 182.403125 17.999219 \n","L 182.403125 46.355469 \n","Q 182.403125 48.355469 184.403125 48.355469 \n","z\n","\" style=\"fill: #ffffff; opacity: 0.8; stroke: #cccccc; stroke-linejoin: miter\"/>\n","    </g>\n","    <g id=\"line2d_67\">\n","     <path d=\"M 186.403125 24.097656 \n","L 196.403125 24.097656 \n","L 206.403125 24.097656 \n","\" style=\"fill: none; stroke: #1f77b4; stroke-width: 1.5; stroke-linecap: square\"/>\n","    </g>\n","    <g id=\"text_15\">\n","     <!-- train -->\n","     <g transform=\"translate(214.403125 27.597656)scale(0.1 -0.1)\">\n","      <defs>\n","       <path id=\"DejaVuSans-74\" d=\"M 1172 4494 \n","L 1172 3500 \n","L 2356 3500 \n","L 2356 3053 \n","L 1172 3053 \n","L 1172 1153 \n","Q 1172 725 1289 603 \n","Q 1406 481 1766 481 \n","L 2356 481 \n","L 2356 0 \n","L 1766 0 \n","Q 1100 0 847 248 \n","Q 594 497 594 1153 \n","L 594 3053 \n","L 172 3053 \n","L 172 3500 \n","L 594 3500 \n","L 594 4494 \n","L 1172 4494 \n","z\n","\" transform=\"scale(0.015625)\"/>\n","       <path id=\"DejaVuSans-72\" d=\"M 2631 2963 \n","Q 2534 3019 2420 3045 \n","Q 2306 3072 2169 3072 \n","Q 1681 3072 1420 2755 \n","Q 1159 2438 1159 1844 \n","L 1159 0 \n","L 581 0 \n","L 581 3500 \n","L 1159 3500 \n","L 1159 2956 \n","Q 1341 3275 1631 3429 \n","Q 1922 3584 2338 3584 \n","Q 2397 3584 2469 3576 \n","Q 2541 3569 2628 3553 \n","L 2631 2963 \n","z\n","\" transform=\"scale(0.015625)\"/>\n","       <path id=\"DejaVuSans-61\" d=\"M 2194 1759 \n","Q 1497 1759 1228 1600 \n","Q 959 1441 959 1056 \n","Q 959 750 1161 570 \n","Q 1363 391 1709 391 \n","Q 2188 391 2477 730 \n","Q 2766 1069 2766 1631 \n","L 2766 1759 \n","L 2194 1759 \n","z\n","M 3341 1997 \n","L 3341 0 \n","L 2766 0 \n","L 2766 531 \n","Q 2569 213 2275 61 \n","Q 1981 -91 1556 -91 \n","Q 1019 -91 701 211 \n","Q 384 513 384 1019 \n","Q 384 1609 779 1909 \n","Q 1175 2209 1959 2209 \n","L 2766 2209 \n","L 2766 2266 \n","Q 2766 2663 2505 2880 \n","Q 2244 3097 1772 3097 \n","Q 1472 3097 1187 3025 \n","Q 903 2953 641 2809 \n","L 641 3341 \n","Q 956 3463 1253 3523 \n","Q 1550 3584 1831 3584 \n","Q 2591 3584 2966 3190 \n","Q 3341 2797 3341 1997 \n","z\n","\" transform=\"scale(0.015625)\"/>\n","       <path id=\"DejaVuSans-69\" d=\"M 603 3500 \n","L 1178 3500 \n","L 1178 0 \n","L 603 0 \n","L 603 3500 \n","z\n","M 603 4863 \n","L 1178 4863 \n","L 1178 4134 \n","L 603 4134 \n","L 603 4863 \n","z\n","\" transform=\"scale(0.015625)\"/>\n","       <path id=\"DejaVuSans-6e\" d=\"M 3513 2113 \n","L 3513 0 \n","L 2938 0 \n","L 2938 2094 \n","Q 2938 2591 2744 2837 \n","Q 2550 3084 2163 3084 \n","Q 1697 3084 1428 2787 \n","Q 1159 2491 1159 1978 \n","L 1159 0 \n","L 581 0 \n","L 581 3500 \n","L 1159 3500 \n","L 1159 2956 \n","Q 1366 3272 1645 3428 \n","Q 1925 3584 2291 3584 \n","Q 2894 3584 3203 3211 \n","Q 3513 2838 3513 2113 \n","z\n","\" transform=\"scale(0.015625)\"/>\n","      </defs>\n","      <use xlink:href=\"#DejaVuSans-74\"/>\n","      <use xlink:href=\"#DejaVuSans-72\" x=\"39.208984\"/>\n","      <use xlink:href=\"#DejaVuSans-61\" x=\"80.322266\"/>\n","      <use xlink:href=\"#DejaVuSans-69\" x=\"141.601562\"/>\n","      <use xlink:href=\"#DejaVuSans-6e\" x=\"169.384766\"/>\n","     </g>\n","    </g>\n","    <g id=\"line2d_68\">\n","     <path d=\"M 186.403125 38.775781 \n","L 196.403125 38.775781 \n","L 206.403125 38.775781 \n","\" style=\"fill: none; stroke-dasharray: 5.55,2.4; stroke-dashoffset: 0; stroke: #bf00bf; stroke-width: 1.5\"/>\n","    </g>\n","    <g id=\"text_16\">\n","     <!-- test -->\n","     <g transform=\"translate(214.403125 42.275781)scale(0.1 -0.1)\">\n","      <use xlink:href=\"#DejaVuSans-74\"/>\n","      <use xlink:href=\"#DejaVuSans-65\" x=\"39.208984\"/>\n","      <use xlink:href=\"#DejaVuSans-73\" x=\"100.732422\"/>\n","      <use xlink:href=\"#DejaVuSans-74\" x=\"152.832031\"/>\n","     </g>\n","    </g>\n","   </g>\n","  </g>\n"," </g>\n"," <defs>\n","  <clipPath id=\"p44b2c4c390\">\n","   <rect x=\"51.378125\" y=\"10.999219\" width=\"195.3\" height=\"135.9\"/>\n","  </clipPath>\n"," </defs>\n","</svg>\n"],"text/plain":["<Figure size 252x180 with 1 Axes>"]},"metadata":{"needs_background":"light"},"output_type":"display_data"}],"source":["# 从多项式特征中选取所有维度\n","train(poly_features[:n_train, :], poly_features[n_train:, :],\n","      labels[:n_train], labels[n_train:], num_epochs=1500)"],"id":"ac04a8f2"},{"cell_type":"markdown","source":[],"metadata":{"id":"i4PWuxo3hFGm"},"id":"i4PWuxo3hFGm"},{"cell_type":"markdown","metadata":{"origin_pos":0,"id":"sjyhgqsqhFj0"},"source":["# 权重衰减\n","\n","\n","前一节我们描述了过拟合的问题，本节我们将介绍一些正则化模型的技术。\n","我们总是可以通过去收集更多的训练数据来缓解过拟合。\n","但这可能成本很高，耗时颇多，或者完全超出我们的控制，因而在短期内不可能做到。\n","假设我们已经拥有尽可能多的高质量数据，我们便可以将重点放在正则化技术上。\n","\n","回想一下，在多项式回归的例子（ ）中，\n","我们可以通过调整拟合多项式的阶数来限制模型的容量。\n","实际上，限制特征的数量是缓解过拟合的一种常用技术。\n","然而，简单地丢弃特征对于这项工作来说可能过于生硬。\n","我们继续思考多项式回归的例子，考虑高维输入可能发生的情况。\n","多项式对多变量数据的自然扩展称为*单项式*（monomials），\n","也可以说是变量幂的乘积。\n","单项式的阶数是幂的和。\n","例如，$x_1^2 x_2$和$x_3 x_5^2$都是3次单项式。\n","\n","注意，随着阶数$d$的增长，带有阶数$d$的项数迅速增加。 \n","给定$k$个变量，阶数为$d$的项的个数为\n","${k - 1 + d} \\choose {k - 1}$，即$C^{k-1}_{k-1+d} = \\frac{(k-1+d)!}{(d)!(k-1)!}$。\n","因此即使是阶数上的微小变化，比如从$2$到$3$，也会显著增加我们模型的复杂性。\n","仅仅通过简单的限制特征数量（在多项式回归中体现为限制阶数），可能仍然使模型在过简单和过复杂中徘徊，\n","我们需要一个更细粒度的工具来调整函数的复杂性，使其达到一个合适的平衡位置。\n","## 范数与权重衰减\n","\n","我们已经描述了$L_2$范数和$L_1$范数，\n","它们是更为一般的$L_p$范数的特殊情况。\n","(~~权重衰减是最广泛使用的正则化的技术之一~~)\n","在训练参数化机器学习模型时，\n","*权重衰减*（weight decay）是最广泛使用的正则化的技术之一，\n","它通常也被称为$L_2$*正则化*。\n","这项技术通过函数与零的距离来衡量函数的复杂度，\n","因为在所有函数$f$中，函数$f = 0$（所有输入都得到值$0$）\n","在某种意义上是最简单的。\n","但是我们应该如何精确地测量一个函数和零之间的距离呢？\n","没有一个正确的答案。\n","事实上，函数分析和巴拿赫空间理论的研究，都在致力于回答这个问题。\n","\n","一种简单的方法是通过线性函数\n","$f(\\mathbf{x}) = \\mathbf{w}^\\top \\mathbf{x}$\n","中的权重向量的某个范数来度量其复杂性，\n","例如$\\| \\mathbf{w} \\|^2$。\n","要保证权重向量比较小，\n","最常用方法是将其范数作为惩罚项加到最小化损失的问题中。\n","将原来的训练目标*最小化训练标签上的预测损失*，\n","调整为*最小化预测损失和惩罚项之和*。\n","现在，如果我们的权重向量增长的太大，\n","我们的学习算法可能会更集中于最小化权重范数$\\| \\mathbf{w} \\|^2$。\n","这正是我们想要的。\n","让我们回顾一下中的线性回归例子。\n","我们的损失由下式给出：\n","\n","$$L(\\mathbf{w}, b) = \\frac{1}{n}\\sum_{i=1}^n \\frac{1}{2}\\left(\\mathbf{w}^\\top \\mathbf{x}^{(i)} + b - y^{(i)}\\right)^2.$$\n","\n","回想一下，$\\mathbf{x}^{(i)}$是样本$i$的特征，\n","$y^{(i)}$是样本$i$的标签，\n","$(\\mathbf{w}, b)$是权重和偏置参数。\n","为了惩罚权重向量的大小，\n","我们必须以某种方式在损失函数中添加$\\| \\mathbf{w} \\|^2$，\n","但是模型应该如何平衡这个新的额外惩罚的损失？\n","实际上，我们通过*正则化常数*$\\lambda$来描述这种权衡，\n","这是一个非负超参数，我们使用验证数据拟合：\n","\n","$$L(\\mathbf{w}, b) + \\frac{\\lambda}{2} \\|\\mathbf{w}\\|^2,$$\n","\n","对于$\\lambda = 0$，我们恢复了原来的损失函数。\n","对于$\\lambda > 0$，我们限制$\\| \\mathbf{w} \\|$的大小。\n","这里我们仍然除以$2$：当我们取一个二次函数的导数时，\n","$2$和$1/2$会抵消，以确保更新表达式看起来既漂亮又简单。\n","你可能会想知道为什么我们使用平方范数而不是标准范数（即欧几里得距离）？\n","我们这样做是为了便于计算。\n","通过平方$L_2$范数，我们去掉平方根，留下权重向量每个分量的平方和。\n","这使得惩罚的导数很容易计算：导数的和等于和的导数。\n","\n","此外，你可能会问为什么我们首先使用$L_2$范数，而不是$L_1$范数。\n","事实上，这个选择在整个统计领域中都是有效的和受欢迎的。\n","$L_2$正则化线性模型构成经典的*岭回归*（ridge regression）算法，\n","$L_1$正则化线性回归是统计学中类似的基本模型，\n","通常被称为*套索回归*（lasso regression）。\n","使用$L_2$范数的一个原因是它对权重向量的大分量施加了巨大的惩罚。\n","这使得我们的学习算法偏向于在大量特征上均匀分布权重的模型。\n","在实践中，这可能使它们对单个变量中的观测误差更为稳定。\n","相比之下，$L_1$惩罚会导致模型将权重集中在一小部分特征上，\n","而将其他权重清除为零。\n","这称为*特征选择*（feature selection），这可能是其他场景下需要的。\n","\n","使用与中的相同符号，\n","$L_2$正则化回归的小批量随机梯度下降更新如下式：\n","\n","$$\n","\\begin{aligned}\n","\\mathbf{w} & \\leftarrow \\left(1- \\eta\\lambda \\right) \\mathbf{w} - \\frac{\\eta}{|\\mathcal{B}|} \\sum_{i \\in \\mathcal{B}} \\mathbf{x}^{(i)} \\left(\\mathbf{w}^\\top \\mathbf{x}^{(i)} + b - y^{(i)}\\right).\n","\\end{aligned}\n","$$\n","\n","根据之前章节所讲的，我们根据估计值与观测值之间的差异来更新$\\mathbf{w}$。\n","然而，我们同时也在试图将$\\mathbf{w}$的大小缩小到零。\n","这就是为什么这种方法有时被称为*权重衰减*。\n","我们仅考虑惩罚项，优化算法在训练的每一步*衰减*权重。\n","与特征选择相比，权重衰减为我们提供了一种连续的机制来调整函数的复杂度。\n","较小的$\\lambda$值对应较少约束的$\\mathbf{w}$，\n","而较大的$\\lambda$值对$\\mathbf{w}$的约束更大。\n","\n","是否对相应的偏置$b^2$进行惩罚在不同的实践中会有所不同，\n","在神经网络的不同层中也会有所不同。\n","通常，网络输出层的偏置项不会被正则化。\n","\n","通常 $\\eta\\lambda < 1$ ，在深度学习中通常叫做权重衰退\n","\n","## 高维线性回归\n","\n","我们通过一个简单的例子来演示权重衰减。\n"],"id":"sjyhgqsqhFj0"},{"cell_type":"code","execution_count":null,"metadata":{"execution":{"iopub.execute_input":"2022-07-31T02:22:47.330465Z","iopub.status.busy":"2022-07-31T02:22:47.329932Z","iopub.status.idle":"2022-07-31T02:22:49.309063Z","shell.execute_reply":"2022-07-31T02:22:49.308346Z"},"origin_pos":2,"tab":["pytorch"],"id":"AW5_eW1EhFj1"},"outputs":[],"source":["%matplotlib inline\n","import torch\n","from torch import nn\n","from d2l import torch as d2l"],"id":"AW5_eW1EhFj1"},{"cell_type":"markdown","metadata":{"origin_pos":4,"id":"ST2dH7DghFj1"},"source":["首先，我们[**像以前一样生成一些数据**]，生成公式如下：\n","\n","(**$$y = 0.05 + \\sum_{i = 1}^d 0.01 x_i + \\epsilon \\text{ where }\n","\\epsilon \\sim \\mathcal{N}(0, 0.01^2).$$**)\n","\n","我们选择标签是关于输入的线性函数。\n","标签同时被均值为0，标准差为0.01高斯噪声破坏。\n","为了使过拟合的效果更加明显，我们可以将问题的维数增加到$d = 200$，\n","并使用一个只包含20个样本的小训练集。\n"],"id":"ST2dH7DghFj1"},{"cell_type":"code","execution_count":null,"metadata":{"execution":{"iopub.execute_input":"2022-07-31T02:22:49.312954Z","iopub.status.busy":"2022-07-31T02:22:49.312444Z","iopub.status.idle":"2022-07-31T02:22:49.319046Z","shell.execute_reply":"2022-07-31T02:22:49.318390Z"},"origin_pos":5,"tab":["pytorch"],"id":"0zRmfl_OhFj1"},"outputs":[],"source":["n_train, n_test, num_inputs, batch_size = 20, 100, 200, 5\n","true_w, true_b = torch.ones((num_inputs, 1)) * 0.01, 0.05\n","train_data = d2l.synthetic_data(true_w, true_b, n_train)\n","train_iter = d2l.load_array(train_data, batch_size)\n","test_data = d2l.synthetic_data(true_w, true_b, n_test)\n","test_iter = d2l.load_array(test_data, batch_size, is_train=False)"],"id":"0zRmfl_OhFj1"},{"cell_type":"markdown","metadata":{"origin_pos":6,"id":"a8M517mzhFj1"},"source":["## 从零开始实现\n","\n","下面我们将从头开始实现权重衰减，只需将$L_2$的平方惩罚添加到原始目标函数中。\n","\n","### [**初始化模型参数**]\n","\n","首先，我们将定义一个函数来随机初始化模型参数。\n"],"id":"a8M517mzhFj1"},{"cell_type":"code","execution_count":null,"metadata":{"execution":{"iopub.execute_input":"2022-07-31T02:22:49.323345Z","iopub.status.busy":"2022-07-31T02:22:49.322872Z","iopub.status.idle":"2022-07-31T02:22:49.327215Z","shell.execute_reply":"2022-07-31T02:22:49.326404Z"},"origin_pos":8,"tab":["pytorch"],"id":"fibLeaqThFj1"},"outputs":[],"source":["def init_params():\n","    w = torch.normal(0, 1, size=(num_inputs, 1), requires_grad=True)\n","    b = torch.zeros(1, requires_grad=True)\n","    return [w, b]"],"id":"fibLeaqThFj1"},{"cell_type":"markdown","metadata":{"origin_pos":10,"id":"NqKPx-9WhFj1"},"source":["### (**定义$L_2$范数惩罚**)\n","\n","实现这一惩罚最方便的方法是对所有项求平方后并将它们求和。\n"],"id":"NqKPx-9WhFj1"},{"cell_type":"code","execution_count":null,"metadata":{"execution":{"iopub.execute_input":"2022-07-31T02:22:49.331227Z","iopub.status.busy":"2022-07-31T02:22:49.330783Z","iopub.status.idle":"2022-07-31T02:22:49.334599Z","shell.execute_reply":"2022-07-31T02:22:49.333840Z"},"origin_pos":12,"tab":["pytorch"],"id":"SIrWIImRhFj2"},"outputs":[],"source":["def l2_penalty(w):\n","    return torch.sum(w.pow(2)) / 2"],"id":"SIrWIImRhFj2"},{"cell_type":"markdown","metadata":{"origin_pos":14,"id":"X8doqvkjhFj2"},"source":["### [**定义训练代码实现**]\n","\n","下面的代码将模型拟合训练数据集，并在测试数据集上进行评估。\n","从 :numref:`chap_linear`以来，线性网络和平方损失没有变化，\n","所以我们通过`d2l.linreg`和`d2l.squared_loss`导入它们。\n","唯一的变化是损失现在包括了惩罚项。\n"],"id":"X8doqvkjhFj2"},{"cell_type":"code","execution_count":null,"metadata":{"execution":{"iopub.execute_input":"2022-07-31T02:22:49.338662Z","iopub.status.busy":"2022-07-31T02:22:49.338197Z","iopub.status.idle":"2022-07-31T02:22:49.344902Z","shell.execute_reply":"2022-07-31T02:22:49.344113Z"},"origin_pos":16,"tab":["pytorch"],"id":"SFpfGvLqhFj2"},"outputs":[],"source":["def train(lambd):\n","    w, b = init_params()\n","    net, loss = lambda X: d2l.linreg(X, w, b), d2l.squared_loss\n","    num_epochs, lr = 100, 0.003\n","    animator = d2l.Animator(xlabel='epochs', ylabel='loss', yscale='log',\n","                            xlim=[5, num_epochs], legend=['train', 'test'])\n","    for epoch in range(num_epochs):\n","        for X, y in train_iter:\n","            # 增加了L2范数惩罚项，\n","            # 广播机制使l2_penalty(w)成为一个长度为batch_size的向量\n","            l = loss(net(X), y) + lambd * l2_penalty(w)\n","            l.sum().backward()\n","            d2l.sgd([w, b], lr, batch_size)\n","        if (epoch + 1) % 5 == 0:\n","            animator.add(epoch + 1, (d2l.evaluate_loss(net, train_iter, loss),\n","                                     d2l.evaluate_loss(net, test_iter, loss)))\n","    print('w的L2范数是：', torch.norm(w).item())"],"id":"SFpfGvLqhFj2"},{"cell_type":"markdown","metadata":{"origin_pos":18,"id":"y_-4ySaVhFj2"},"source":["### [**忽略正则化直接训练**]\n","\n","我们现在用`lambd = 0`禁用权重衰减后运行这个代码。\n","注意，这里训练误差有了减少，但测试误差没有减少，\n","这意味着出现了严重的过拟合。\n"],"id":"y_-4ySaVhFj2"},{"cell_type":"code","execution_count":null,"metadata":{"execution":{"iopub.execute_input":"2022-07-31T02:22:49.348006Z","iopub.status.busy":"2022-07-31T02:22:49.347416Z","iopub.status.idle":"2022-07-31T02:22:55.832837Z","shell.execute_reply":"2022-07-31T02:22:55.832068Z"},"origin_pos":19,"tab":["pytorch"],"outputId":"7d0c9c39-cae7-4882-8415-f1c012e36c27","id":"-FwnI4bkhFj2"},"outputs":[{"name":"stdout","output_type":"stream","text":["w的L2范数是： 12.791678428649902\n"]},{"data":{"image/svg+xml":["<?xml version=\"1.0\" encoding=\"utf-8\" standalone=\"no\"?>\n","<!DOCTYPE svg PUBLIC \"-//W3C//DTD SVG 1.1//EN\"\n","  \"http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd\">\n","<svg xmlns:xlink=\"http://www.w3.org/1999/xlink\" width=\"263.421875pt\" height=\"180.65625pt\" viewBox=\"0 0 263.421875 180.65625\" xmlns=\"http://www.w3.org/2000/svg\" version=\"1.1\">\n"," <metadata>\n","  <rdf:RDF xmlns:dc=\"http://purl.org/dc/elements/1.1/\" xmlns:cc=\"http://creativecommons.org/ns#\" xmlns:rdf=\"http://www.w3.org/1999/02/22-rdf-syntax-ns#\">\n","   <cc:Work>\n","    <dc:type rdf:resource=\"http://purl.org/dc/dcmitype/StillImage\"/>\n","    <dc:date>2022-07-31T02:22:55.759832</dc:date>\n","    <dc:format>image/svg+xml</dc:format>\n","    <dc:creator>\n","     <cc:Agent>\n","      <dc:title>Matplotlib v3.5.1, https://matplotlib.org/</dc:title>\n","     </cc:Agent>\n","    </dc:creator>\n","   </cc:Work>\n","  </rdf:RDF>\n"," </metadata>\n"," <defs>\n","  <style type=\"text/css\">*{stroke-linejoin: round; stroke-linecap: butt}</style>\n"," </defs>\n"," <g id=\"figure_1\">\n","  <g id=\"patch_1\">\n","   <path d=\"M 0 180.65625 \n","L 263.421875 180.65625 \n","L 263.421875 0 \n","L 0 0 \n","L 0 180.65625 \n","z\n","\" style=\"fill: none\"/>\n","  </g>\n","  <g id=\"axes_1\">\n","   <g id=\"patch_2\">\n","    <path d=\"M 51.378125 143.1 \n","L 246.678125 143.1 \n","L 246.678125 7.2 \n","L 51.378125 7.2 \n","z\n","\" style=\"fill: #ffffff\"/>\n","   </g>\n","   <g id=\"matplotlib.axis_1\">\n","    <g id=\"xtick_1\">\n","     <g id=\"line2d_1\">\n","      <path d=\"M 82.214967 143.1 \n","L 82.214967 7.2 \n","\" clip-path=\"url(#paf1508e7f6)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n","     </g>\n","     <g id=\"line2d_2\">\n","      <defs>\n","       <path id=\"m3a97404132\" d=\"M 0 0 \n","L 0 3.5 \n","\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n","      </defs>\n","      <g>\n","       <use xlink:href=\"#m3a97404132\" x=\"82.214967\" y=\"143.1\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n","      </g>\n","     </g>\n","     <g id=\"text_1\">\n","      <!-- 20 -->\n","      <g transform=\"translate(75.852467 157.698438)scale(0.1 -0.1)\">\n","       <defs>\n","        <path id=\"DejaVuSans-32\" d=\"M 1228 531 \n","L 3431 531 \n","L 3431 0 \n","L 469 0 \n","L 469 531 \n","Q 828 903 1448 1529 \n","Q 2069 2156 2228 2338 \n","Q 2531 2678 2651 2914 \n","Q 2772 3150 2772 3378 \n","Q 2772 3750 2511 3984 \n","Q 2250 4219 1831 4219 \n","Q 1534 4219 1204 4116 \n","Q 875 4013 500 3803 \n","L 500 4441 \n","Q 881 4594 1212 4672 \n","Q 1544 4750 1819 4750 \n","Q 2544 4750 2975 4387 \n","Q 3406 4025 3406 3419 \n","Q 3406 3131 3298 2873 \n","Q 3191 2616 2906 2266 \n","Q 2828 2175 2409 1742 \n","Q 1991 1309 1228 531 \n","z\n","\" transform=\"scale(0.015625)\"/>\n","        <path id=\"DejaVuSans-30\" d=\"M 2034 4250 \n","Q 1547 4250 1301 3770 \n","Q 1056 3291 1056 2328 \n","Q 1056 1369 1301 889 \n","Q 1547 409 2034 409 \n","Q 2525 409 2770 889 \n","Q 3016 1369 3016 2328 \n","Q 3016 3291 2770 3770 \n","Q 2525 4250 2034 4250 \n","z\n","M 2034 4750 \n","Q 2819 4750 3233 4129 \n","Q 3647 3509 3647 2328 \n","Q 3647 1150 3233 529 \n","Q 2819 -91 2034 -91 \n","Q 1250 -91 836 529 \n","Q 422 1150 422 2328 \n","Q 422 3509 836 4129 \n","Q 1250 4750 2034 4750 \n","z\n","\" transform=\"scale(0.015625)\"/>\n","       </defs>\n","       <use xlink:href=\"#DejaVuSans-32\"/>\n","       <use xlink:href=\"#DejaVuSans-30\" x=\"63.623047\"/>\n","      </g>\n","     </g>\n","    </g>\n","    <g id=\"xtick_2\">\n","     <g id=\"line2d_3\">\n","      <path d=\"M 123.330757 143.1 \n","L 123.330757 7.2 \n","\" clip-path=\"url(#paf1508e7f6)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n","     </g>\n","     <g id=\"line2d_4\">\n","      <g>\n","       <use xlink:href=\"#m3a97404132\" x=\"123.330757\" y=\"143.1\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n","      </g>\n","     </g>\n","     <g id=\"text_2\">\n","      <!-- 40 -->\n","      <g transform=\"translate(116.968257 157.698438)scale(0.1 -0.1)\">\n","       <defs>\n","        <path id=\"DejaVuSans-34\" d=\"M 2419 4116 \n","L 825 1625 \n","L 2419 1625 \n","L 2419 4116 \n","z\n","M 2253 4666 \n","L 3047 4666 \n","L 3047 1625 \n","L 3713 1625 \n","L 3713 1100 \n","L 3047 1100 \n","L 3047 0 \n","L 2419 0 \n","L 2419 1100 \n","L 313 1100 \n","L 313 1709 \n","L 2253 4666 \n","z\n","\" transform=\"scale(0.015625)\"/>\n","       </defs>\n","       <use xlink:href=\"#DejaVuSans-34\"/>\n","       <use xlink:href=\"#DejaVuSans-30\" x=\"63.623047\"/>\n","      </g>\n","     </g>\n","    </g>\n","    <g id=\"xtick_3\">\n","     <g id=\"line2d_5\">\n","      <path d=\"M 164.446546 143.1 \n","L 164.446546 7.2 \n","\" clip-path=\"url(#paf1508e7f6)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n","     </g>\n","     <g id=\"line2d_6\">\n","      <g>\n","       <use xlink:href=\"#m3a97404132\" x=\"164.446546\" y=\"143.1\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n","      </g>\n","     </g>\n","     <g id=\"text_3\">\n","      <!-- 60 -->\n","      <g transform=\"translate(158.084046 157.698438)scale(0.1 -0.1)\">\n","       <defs>\n","        <path id=\"DejaVuSans-36\" d=\"M 2113 2584 \n","Q 1688 2584 1439 2293 \n","Q 1191 2003 1191 1497 \n","Q 1191 994 1439 701 \n","Q 1688 409 2113 409 \n","Q 2538 409 2786 701 \n","Q 3034 994 3034 1497 \n","Q 3034 2003 2786 2293 \n","Q 2538 2584 2113 2584 \n","z\n","M 3366 4563 \n","L 3366 3988 \n","Q 3128 4100 2886 4159 \n","Q 2644 4219 2406 4219 \n","Q 1781 4219 1451 3797 \n","Q 1122 3375 1075 2522 \n","Q 1259 2794 1537 2939 \n","Q 1816 3084 2150 3084 \n","Q 2853 3084 3261 2657 \n","Q 3669 2231 3669 1497 \n","Q 3669 778 3244 343 \n","Q 2819 -91 2113 -91 \n","Q 1303 -91 875 529 \n","Q 447 1150 447 2328 \n","Q 447 3434 972 4092 \n","Q 1497 4750 2381 4750 \n","Q 2619 4750 2861 4703 \n","Q 3103 4656 3366 4563 \n","z\n","\" transform=\"scale(0.015625)\"/>\n","       </defs>\n","       <use xlink:href=\"#DejaVuSans-36\"/>\n","       <use xlink:href=\"#DejaVuSans-30\" x=\"63.623047\"/>\n","      </g>\n","     </g>\n","    </g>\n","    <g id=\"xtick_4\">\n","     <g id=\"line2d_7\">\n","      <path d=\"M 205.562336 143.1 \n","L 205.562336 7.2 \n","\" clip-path=\"url(#paf1508e7f6)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n","     </g>\n","     <g id=\"line2d_8\">\n","      <g>\n","       <use xlink:href=\"#m3a97404132\" x=\"205.562336\" y=\"143.1\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n","      </g>\n","     </g>\n","     <g id=\"text_4\">\n","      <!-- 80 -->\n","      <g transform=\"translate(199.199836 157.698438)scale(0.1 -0.1)\">\n","       <defs>\n","        <path id=\"DejaVuSans-38\" d=\"M 2034 2216 \n","Q 1584 2216 1326 1975 \n","Q 1069 1734 1069 1313 \n","Q 1069 891 1326 650 \n","Q 1584 409 2034 409 \n","Q 2484 409 2743 651 \n","Q 3003 894 3003 1313 \n","Q 3003 1734 2745 1975 \n","Q 2488 2216 2034 2216 \n","z\n","M 1403 2484 \n","Q 997 2584 770 2862 \n","Q 544 3141 544 3541 \n","Q 544 4100 942 4425 \n","Q 1341 4750 2034 4750 \n","Q 2731 4750 3128 4425 \n","Q 3525 4100 3525 3541 \n","Q 3525 3141 3298 2862 \n","Q 3072 2584 2669 2484 \n","Q 3125 2378 3379 2068 \n","Q 3634 1759 3634 1313 \n","Q 3634 634 3220 271 \n","Q 2806 -91 2034 -91 \n","Q 1263 -91 848 271 \n","Q 434 634 434 1313 \n","Q 434 1759 690 2068 \n","Q 947 2378 1403 2484 \n","z\n","M 1172 3481 \n","Q 1172 3119 1398 2916 \n","Q 1625 2713 2034 2713 \n","Q 2441 2713 2670 2916 \n","Q 2900 3119 2900 3481 \n","Q 2900 3844 2670 4047 \n","Q 2441 4250 2034 4250 \n","Q 1625 4250 1398 4047 \n","Q 1172 3844 1172 3481 \n","z\n","\" transform=\"scale(0.015625)\"/>\n","       </defs>\n","       <use xlink:href=\"#DejaVuSans-38\"/>\n","       <use xlink:href=\"#DejaVuSans-30\" x=\"63.623047\"/>\n","      </g>\n","     </g>\n","    </g>\n","    <g id=\"xtick_5\">\n","     <g id=\"line2d_9\">\n","      <path d=\"M 246.678125 143.1 \n","L 246.678125 7.2 \n","\" clip-path=\"url(#paf1508e7f6)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n","     </g>\n","     <g id=\"line2d_10\">\n","      <g>\n","       <use xlink:href=\"#m3a97404132\" x=\"246.678125\" y=\"143.1\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n","      </g>\n","     </g>\n","     <g id=\"text_5\">\n","      <!-- 100 -->\n","      <g transform=\"translate(237.134375 157.698438)scale(0.1 -0.1)\">\n","       <defs>\n","        <path id=\"DejaVuSans-31\" d=\"M 794 531 \n","L 1825 531 \n","L 1825 4091 \n","L 703 3866 \n","L 703 4441 \n","L 1819 4666 \n","L 2450 4666 \n","L 2450 531 \n","L 3481 531 \n","L 3481 0 \n","L 794 0 \n","L 794 531 \n","z\n","\" transform=\"scale(0.015625)\"/>\n","       </defs>\n","       <use xlink:href=\"#DejaVuSans-31\"/>\n","       <use xlink:href=\"#DejaVuSans-30\" x=\"63.623047\"/>\n","       <use xlink:href=\"#DejaVuSans-30\" x=\"127.246094\"/>\n","      </g>\n","     </g>\n","    </g>\n","    <g id=\"text_6\">\n","     <!-- epochs -->\n","     <g transform=\"translate(131.195312 171.376563)scale(0.1 -0.1)\">\n","      <defs>\n","       <path id=\"DejaVuSans-65\" d=\"M 3597 1894 \n","L 3597 1613 \n","L 953 1613 \n","Q 991 1019 1311 708 \n","Q 1631 397 2203 397 \n","Q 2534 397 2845 478 \n","Q 3156 559 3463 722 \n","L 3463 178 \n","Q 3153 47 2828 -22 \n","Q 2503 -91 2169 -91 \n","Q 1331 -91 842 396 \n","Q 353 884 353 1716 \n","Q 353 2575 817 3079 \n","Q 1281 3584 2069 3584 \n","Q 2775 3584 3186 3129 \n","Q 3597 2675 3597 1894 \n","z\n","M 3022 2063 \n","Q 3016 2534 2758 2815 \n","Q 2500 3097 2075 3097 \n","Q 1594 3097 1305 2825 \n","Q 1016 2553 972 2059 \n","L 3022 2063 \n","z\n","\" transform=\"scale(0.015625)\"/>\n","       <path id=\"DejaVuSans-70\" d=\"M 1159 525 \n","L 1159 -1331 \n","L 581 -1331 \n","L 581 3500 \n","L 1159 3500 \n","L 1159 2969 \n","Q 1341 3281 1617 3432 \n","Q 1894 3584 2278 3584 \n","Q 2916 3584 3314 3078 \n","Q 3713 2572 3713 1747 \n","Q 3713 922 3314 415 \n","Q 2916 -91 2278 -91 \n","Q 1894 -91 1617 61 \n","Q 1341 213 1159 525 \n","z\n","M 3116 1747 \n","Q 3116 2381 2855 2742 \n","Q 2594 3103 2138 3103 \n","Q 1681 3103 1420 2742 \n","Q 1159 2381 1159 1747 \n","Q 1159 1113 1420 752 \n","Q 1681 391 2138 391 \n","Q 2594 391 2855 752 \n","Q 3116 1113 3116 1747 \n","z\n","\" transform=\"scale(0.015625)\"/>\n","       <path id=\"DejaVuSans-6f\" d=\"M 1959 3097 \n","Q 1497 3097 1228 2736 \n","Q 959 2375 959 1747 \n","Q 959 1119 1226 758 \n","Q 1494 397 1959 397 \n","Q 2419 397 2687 759 \n","Q 2956 1122 2956 1747 \n","Q 2956 2369 2687 2733 \n","Q 2419 3097 1959 3097 \n","z\n","M 1959 3584 \n","Q 2709 3584 3137 3096 \n","Q 3566 2609 3566 1747 \n","Q 3566 888 3137 398 \n","Q 2709 -91 1959 -91 \n","Q 1206 -91 779 398 \n","Q 353 888 353 1747 \n","Q 353 2609 779 3096 \n","Q 1206 3584 1959 3584 \n","z\n","\" transform=\"scale(0.015625)\"/>\n","       <path id=\"DejaVuSans-63\" d=\"M 3122 3366 \n","L 3122 2828 \n","Q 2878 2963 2633 3030 \n","Q 2388 3097 2138 3097 \n","Q 1578 3097 1268 2742 \n","Q 959 2388 959 1747 \n","Q 959 1106 1268 751 \n","Q 1578 397 2138 397 \n","Q 2388 397 2633 464 \n","Q 2878 531 3122 666 \n","L 3122 134 \n","Q 2881 22 2623 -34 \n","Q 2366 -91 2075 -91 \n","Q 1284 -91 818 406 \n","Q 353 903 353 1747 \n","Q 353 2603 823 3093 \n","Q 1294 3584 2113 3584 \n","Q 2378 3584 2631 3529 \n","Q 2884 3475 3122 3366 \n","z\n","\" transform=\"scale(0.015625)\"/>\n","       <path id=\"DejaVuSans-68\" d=\"M 3513 2113 \n","L 3513 0 \n","L 2938 0 \n","L 2938 2094 \n","Q 2938 2591 2744 2837 \n","Q 2550 3084 2163 3084 \n","Q 1697 3084 1428 2787 \n","Q 1159 2491 1159 1978 \n","L 1159 0 \n","L 581 0 \n","L 581 4863 \n","L 1159 4863 \n","L 1159 2956 \n","Q 1366 3272 1645 3428 \n","Q 1925 3584 2291 3584 \n","Q 2894 3584 3203 3211 \n","Q 3513 2838 3513 2113 \n","z\n","\" transform=\"scale(0.015625)\"/>\n","       <path id=\"DejaVuSans-73\" d=\"M 2834 3397 \n","L 2834 2853 \n","Q 2591 2978 2328 3040 \n","Q 2066 3103 1784 3103 \n","Q 1356 3103 1142 2972 \n","Q 928 2841 928 2578 \n","Q 928 2378 1081 2264 \n","Q 1234 2150 1697 2047 \n","L 1894 2003 \n","Q 2506 1872 2764 1633 \n","Q 3022 1394 3022 966 \n","Q 3022 478 2636 193 \n","Q 2250 -91 1575 -91 \n","Q 1294 -91 989 -36 \n","Q 684 19 347 128 \n","L 347 722 \n","Q 666 556 975 473 \n","Q 1284 391 1588 391 \n","Q 1994 391 2212 530 \n","Q 2431 669 2431 922 \n","Q 2431 1156 2273 1281 \n","Q 2116 1406 1581 1522 \n","L 1381 1569 \n","Q 847 1681 609 1914 \n","Q 372 2147 372 2553 \n","Q 372 3047 722 3315 \n","Q 1072 3584 1716 3584 \n","Q 2034 3584 2315 3537 \n","Q 2597 3491 2834 3397 \n","z\n","\" transform=\"scale(0.015625)\"/>\n","      </defs>\n","      <use xlink:href=\"#DejaVuSans-65\"/>\n","      <use xlink:href=\"#DejaVuSans-70\" x=\"61.523438\"/>\n","      <use xlink:href=\"#DejaVuSans-6f\" x=\"125\"/>\n","      <use xlink:href=\"#DejaVuSans-63\" x=\"186.181641\"/>\n","      <use xlink:href=\"#DejaVuSans-68\" x=\"241.162109\"/>\n","      <use xlink:href=\"#DejaVuSans-73\" x=\"304.541016\"/>\n","     </g>\n","    </g>\n","   </g>\n","   <g id=\"matplotlib.axis_2\">\n","    <g id=\"ytick_1\">\n","     <g id=\"line2d_11\">\n","      <path d=\"M 51.378125 123.123527 \n","L 246.678125 123.123527 \n","\" clip-path=\"url(#paf1508e7f6)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n","     </g>\n","     <g id=\"line2d_12\">\n","      <defs>\n","       <path id=\"m35267bf620\" d=\"M 0 0 \n","L -3.5 0 \n","\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n","      </defs>\n","      <g>\n","       <use xlink:href=\"#m35267bf620\" x=\"51.378125\" y=\"123.123527\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n","      </g>\n","     </g>\n","     <g id=\"text_7\">\n","      <!-- $\\mathdefault{10^{-5}}$ -->\n","      <g transform=\"translate(20.878125 126.922746)scale(0.1 -0.1)\">\n","       <defs>\n","        <path id=\"DejaVuSans-2212\" d=\"M 678 2272 \n","L 4684 2272 \n","L 4684 1741 \n","L 678 1741 \n","L 678 2272 \n","z\n","\" transform=\"scale(0.015625)\"/>\n","        <path id=\"DejaVuSans-35\" d=\"M 691 4666 \n","L 3169 4666 \n","L 3169 4134 \n","L 1269 4134 \n","L 1269 2991 \n","Q 1406 3038 1543 3061 \n","Q 1681 3084 1819 3084 \n","Q 2600 3084 3056 2656 \n","Q 3513 2228 3513 1497 \n","Q 3513 744 3044 326 \n","Q 2575 -91 1722 -91 \n","Q 1428 -91 1123 -41 \n","Q 819 9 494 109 \n","L 494 744 \n","Q 775 591 1075 516 \n","Q 1375 441 1709 441 \n","Q 2250 441 2565 725 \n","Q 2881 1009 2881 1497 \n","Q 2881 1984 2565 2268 \n","Q 2250 2553 1709 2553 \n","Q 1456 2553 1204 2497 \n","Q 953 2441 691 2322 \n","L 691 4666 \n","z\n","\" transform=\"scale(0.015625)\"/>\n","       </defs>\n","       <use xlink:href=\"#DejaVuSans-31\" transform=\"translate(0 0.684375)\"/>\n","       <use xlink:href=\"#DejaVuSans-30\" transform=\"translate(63.623047 0.684375)\"/>\n","       <use xlink:href=\"#DejaVuSans-2212\" transform=\"translate(128.203125 38.965625)scale(0.7)\"/>\n","       <use xlink:href=\"#DejaVuSans-35\" transform=\"translate(186.855469 38.965625)scale(0.7)\"/>\n","      </g>\n","     </g>\n","    </g>\n","    <g id=\"ytick_2\">\n","     <g id=\"line2d_13\">\n","      <path d=\"M 51.378125 91.281762 \n","L 246.678125 91.281762 \n","\" clip-path=\"url(#paf1508e7f6)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n","     </g>\n","     <g id=\"line2d_14\">\n","      <g>\n","       <use xlink:href=\"#m35267bf620\" x=\"51.378125\" y=\"91.281762\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n","      </g>\n","     </g>\n","     <g id=\"text_8\">\n","      <!-- $\\mathdefault{10^{-3}}$ -->\n","      <g transform=\"translate(20.878125 95.080981)scale(0.1 -0.1)\">\n","       <defs>\n","        <path id=\"DejaVuSans-33\" d=\"M 2597 2516 \n","Q 3050 2419 3304 2112 \n","Q 3559 1806 3559 1356 \n","Q 3559 666 3084 287 \n","Q 2609 -91 1734 -91 \n","Q 1441 -91 1130 -33 \n","Q 819 25 488 141 \n","L 488 750 \n","Q 750 597 1062 519 \n","Q 1375 441 1716 441 \n","Q 2309 441 2620 675 \n","Q 2931 909 2931 1356 \n","Q 2931 1769 2642 2001 \n","Q 2353 2234 1838 2234 \n","L 1294 2234 \n","L 1294 2753 \n","L 1863 2753 \n","Q 2328 2753 2575 2939 \n","Q 2822 3125 2822 3475 \n","Q 2822 3834 2567 4026 \n","Q 2313 4219 1838 4219 \n","Q 1578 4219 1281 4162 \n","Q 984 4106 628 3988 \n","L 628 4550 \n","Q 988 4650 1302 4700 \n","Q 1616 4750 1894 4750 \n","Q 2613 4750 3031 4423 \n","Q 3450 4097 3450 3541 \n","Q 3450 3153 3228 2886 \n","Q 3006 2619 2597 2516 \n","z\n","\" transform=\"scale(0.015625)\"/>\n","       </defs>\n","       <use xlink:href=\"#DejaVuSans-31\" transform=\"translate(0 0.765625)\"/>\n","       <use xlink:href=\"#DejaVuSans-30\" transform=\"translate(63.623047 0.765625)\"/>\n","       <use xlink:href=\"#DejaVuSans-2212\" transform=\"translate(128.203125 39.046875)scale(0.7)\"/>\n","       <use xlink:href=\"#DejaVuSans-33\" transform=\"translate(186.855469 39.046875)scale(0.7)\"/>\n","      </g>\n","     </g>\n","    </g>\n","    <g id=\"ytick_3\">\n","     <g id=\"line2d_15\">\n","      <path d=\"M 51.378125 59.439996 \n","L 246.678125 59.439996 \n","\" clip-path=\"url(#paf1508e7f6)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n","     </g>\n","     <g id=\"line2d_16\">\n","      <g>\n","       <use xlink:href=\"#m35267bf620\" x=\"51.378125\" y=\"59.439996\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n","      </g>\n","     </g>\n","     <g id=\"text_9\">\n","      <!-- $\\mathdefault{10^{-1}}$ -->\n","      <g transform=\"translate(20.878125 63.239215)scale(0.1 -0.1)\">\n","       <use xlink:href=\"#DejaVuSans-31\" transform=\"translate(0 0.684375)\"/>\n","       <use xlink:href=\"#DejaVuSans-30\" transform=\"translate(63.623047 0.684375)\"/>\n","       <use xlink:href=\"#DejaVuSans-2212\" transform=\"translate(128.203125 38.965625)scale(0.7)\"/>\n","       <use xlink:href=\"#DejaVuSans-31\" transform=\"translate(186.855469 38.965625)scale(0.7)\"/>\n","      </g>\n","     </g>\n","    </g>\n","    <g id=\"ytick_4\">\n","     <g id=\"line2d_17\">\n","      <path d=\"M 51.378125 27.598231 \n","L 246.678125 27.598231 \n","\" clip-path=\"url(#paf1508e7f6)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n","     </g>\n","     <g id=\"line2d_18\">\n","      <g>\n","       <use xlink:href=\"#m35267bf620\" x=\"51.378125\" y=\"27.598231\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n","      </g>\n","     </g>\n","     <g id=\"text_10\">\n","      <!-- $\\mathdefault{10^{1}}$ -->\n","      <g transform=\"translate(26.778125 31.397449)scale(0.1 -0.1)\">\n","       <use xlink:href=\"#DejaVuSans-31\" transform=\"translate(0 0.684375)\"/>\n","       <use xlink:href=\"#DejaVuSans-30\" transform=\"translate(63.623047 0.684375)\"/>\n","       <use xlink:href=\"#DejaVuSans-31\" transform=\"translate(128.203125 38.965625)scale(0.7)\"/>\n","      </g>\n","     </g>\n","    </g>\n","    <g id=\"text_11\">\n","     <!-- loss -->\n","     <g transform=\"translate(14.798437 84.807812)rotate(-90)scale(0.1 -0.1)\">\n","      <defs>\n","       <path id=\"DejaVuSans-6c\" d=\"M 603 4863 \n","L 1178 4863 \n","L 1178 0 \n","L 603 0 \n","L 603 4863 \n","z\n","\" transform=\"scale(0.015625)\"/>\n","      </defs>\n","      <use xlink:href=\"#DejaVuSans-6c\"/>\n","      <use xlink:href=\"#DejaVuSans-6f\" x=\"27.783203\"/>\n","      <use xlink:href=\"#DejaVuSans-73\" x=\"88.964844\"/>\n","      <use xlink:href=\"#DejaVuSans-73\" x=\"141.064453\"/>\n","     </g>\n","    </g>\n","   </g>\n","   <g id=\"line2d_19\">\n","    <path d=\"M 51.378125 26.106104 \n","L 61.657072 35.21742 \n","L 71.93602 43.468449 \n","L 82.214967 51.117298 \n","L 92.493914 58.352648 \n","L 102.772862 65.218746 \n","L 113.051809 71.755366 \n","L 123.330757 77.981748 \n","L 133.609704 83.887567 \n","L 143.888651 89.524081 \n","L 154.167599 94.890499 \n","L 164.446546 100.052997 \n","L 174.725493 105.018093 \n","L 185.004441 109.832648 \n","L 195.283388 114.5259 \n","L 205.562336 119.116875 \n","L 215.841283 123.634023 \n","L 226.12023 128.098305 \n","L 236.399178 132.53015 \n","L 246.678125 136.922727 \n","\" clip-path=\"url(#paf1508e7f6)\" style=\"fill: none; stroke: #1f77b4; stroke-width: 1.5; stroke-linecap: square\"/>\n","   </g>\n","   <g id=\"line2d_20\">\n","    <path d=\"M 51.378125 13.38739 \n","L 61.657072 13.41182 \n","L 71.93602 13.407508 \n","L 82.214967 13.399341 \n","L 92.493914 13.392222 \n","L 102.772862 13.386754 \n","L 113.051809 13.383239 \n","L 123.330757 13.380876 \n","L 133.609704 13.379493 \n","L 143.888651 13.378573 \n","L 154.167599 13.378049 \n","L 164.446546 13.377708 \n","L 174.725493 13.377516 \n","L 185.004441 13.377404 \n","L 195.283388 13.377345 \n","L 205.562336 13.377309 \n","L 215.841283 13.377291 \n","L 226.12023 13.377279 \n","L 236.399178 13.377276 \n","L 246.678125 13.377273 \n","\" clip-path=\"url(#paf1508e7f6)\" style=\"fill: none; stroke-dasharray: 5.55,2.4; stroke-dashoffset: 0; stroke: #bf00bf; stroke-width: 1.5\"/>\n","   </g>\n","   <g id=\"patch_3\">\n","    <path d=\"M 51.378125 143.1 \n","L 51.378125 7.2 \n","\" style=\"fill: none; stroke: #000000; stroke-width: 0.8; stroke-linejoin: miter; stroke-linecap: square\"/>\n","   </g>\n","   <g id=\"patch_4\">\n","    <path d=\"M 246.678125 143.1 \n","L 246.678125 7.2 \n","\" style=\"fill: none; stroke: #000000; stroke-width: 0.8; stroke-linejoin: miter; stroke-linecap: square\"/>\n","   </g>\n","   <g id=\"patch_5\">\n","    <path d=\"M 51.378125 143.1 \n","L 246.678125 143.1 \n","\" style=\"fill: none; stroke: #000000; stroke-width: 0.8; stroke-linejoin: miter; stroke-linecap: square\"/>\n","   </g>\n","   <g id=\"patch_6\">\n","    <path d=\"M 51.378125 7.2 \n","L 246.678125 7.2 \n","\" style=\"fill: none; stroke: #000000; stroke-width: 0.8; stroke-linejoin: miter; stroke-linecap: square\"/>\n","   </g>\n","   <g id=\"legend_1\">\n","    <g id=\"patch_7\">\n","     <path d=\"M 58.378125 138.1 \n","L 113.653125 138.1 \n","Q 115.653125 138.1 115.653125 136.1 \n","L 115.653125 107.74375 \n","Q 115.653125 105.74375 113.653125 105.74375 \n","L 58.378125 105.74375 \n","Q 56.378125 105.74375 56.378125 107.74375 \n","L 56.378125 136.1 \n","Q 56.378125 138.1 58.378125 138.1 \n","z\n","\" style=\"fill: #ffffff; opacity: 0.8; stroke: #cccccc; stroke-linejoin: miter\"/>\n","    </g>\n","    <g id=\"line2d_21\">\n","     <path d=\"M 60.378125 113.842187 \n","L 70.378125 113.842187 \n","L 80.378125 113.842187 \n","\" style=\"fill: none; stroke: #1f77b4; stroke-width: 1.5; stroke-linecap: square\"/>\n","    </g>\n","    <g id=\"text_12\">\n","     <!-- train -->\n","     <g transform=\"translate(88.378125 117.342187)scale(0.1 -0.1)\">\n","      <defs>\n","       <path id=\"DejaVuSans-74\" d=\"M 1172 4494 \n","L 1172 3500 \n","L 2356 3500 \n","L 2356 3053 \n","L 1172 3053 \n","L 1172 1153 \n","Q 1172 725 1289 603 \n","Q 1406 481 1766 481 \n","L 2356 481 \n","L 2356 0 \n","L 1766 0 \n","Q 1100 0 847 248 \n","Q 594 497 594 1153 \n","L 594 3053 \n","L 172 3053 \n","L 172 3500 \n","L 594 3500 \n","L 594 4494 \n","L 1172 4494 \n","z\n","\" transform=\"scale(0.015625)\"/>\n","       <path id=\"DejaVuSans-72\" d=\"M 2631 2963 \n","Q 2534 3019 2420 3045 \n","Q 2306 3072 2169 3072 \n","Q 1681 3072 1420 2755 \n","Q 1159 2438 1159 1844 \n","L 1159 0 \n","L 581 0 \n","L 581 3500 \n","L 1159 3500 \n","L 1159 2956 \n","Q 1341 3275 1631 3429 \n","Q 1922 3584 2338 3584 \n","Q 2397 3584 2469 3576 \n","Q 2541 3569 2628 3553 \n","L 2631 2963 \n","z\n","\" transform=\"scale(0.015625)\"/>\n","       <path id=\"DejaVuSans-61\" d=\"M 2194 1759 \n","Q 1497 1759 1228 1600 \n","Q 959 1441 959 1056 \n","Q 959 750 1161 570 \n","Q 1363 391 1709 391 \n","Q 2188 391 2477 730 \n","Q 2766 1069 2766 1631 \n","L 2766 1759 \n","L 2194 1759 \n","z\n","M 3341 1997 \n","L 3341 0 \n","L 2766 0 \n","L 2766 531 \n","Q 2569 213 2275 61 \n","Q 1981 -91 1556 -91 \n","Q 1019 -91 701 211 \n","Q 384 513 384 1019 \n","Q 384 1609 779 1909 \n","Q 1175 2209 1959 2209 \n","L 2766 2209 \n","L 2766 2266 \n","Q 2766 2663 2505 2880 \n","Q 2244 3097 1772 3097 \n","Q 1472 3097 1187 3025 \n","Q 903 2953 641 2809 \n","L 641 3341 \n","Q 956 3463 1253 3523 \n","Q 1550 3584 1831 3584 \n","Q 2591 3584 2966 3190 \n","Q 3341 2797 3341 1997 \n","z\n","\" transform=\"scale(0.015625)\"/>\n","       <path id=\"DejaVuSans-69\" d=\"M 603 3500 \n","L 1178 3500 \n","L 1178 0 \n","L 603 0 \n","L 603 3500 \n","z\n","M 603 4863 \n","L 1178 4863 \n","L 1178 4134 \n","L 603 4134 \n","L 603 4863 \n","z\n","\" transform=\"scale(0.015625)\"/>\n","       <path id=\"DejaVuSans-6e\" d=\"M 3513 2113 \n","L 3513 0 \n","L 2938 0 \n","L 2938 2094 \n","Q 2938 2591 2744 2837 \n","Q 2550 3084 2163 3084 \n","Q 1697 3084 1428 2787 \n","Q 1159 2491 1159 1978 \n","L 1159 0 \n","L 581 0 \n","L 581 3500 \n","L 1159 3500 \n","L 1159 2956 \n","Q 1366 3272 1645 3428 \n","Q 1925 3584 2291 3584 \n","Q 2894 3584 3203 3211 \n","Q 3513 2838 3513 2113 \n","z\n","\" transform=\"scale(0.015625)\"/>\n","      </defs>\n","      <use xlink:href=\"#DejaVuSans-74\"/>\n","      <use xlink:href=\"#DejaVuSans-72\" x=\"39.208984\"/>\n","      <use xlink:href=\"#DejaVuSans-61\" x=\"80.322266\"/>\n","      <use xlink:href=\"#DejaVuSans-69\" x=\"141.601562\"/>\n","      <use xlink:href=\"#DejaVuSans-6e\" x=\"169.384766\"/>\n","     </g>\n","    </g>\n","    <g id=\"line2d_22\">\n","     <path d=\"M 60.378125 128.520312 \n","L 70.378125 128.520312 \n","L 80.378125 128.520312 \n","\" style=\"fill: none; stroke-dasharray: 5.55,2.4; stroke-dashoffset: 0; stroke: #bf00bf; stroke-width: 1.5\"/>\n","    </g>\n","    <g id=\"text_13\">\n","     <!-- test -->\n","     <g transform=\"translate(88.378125 132.020312)scale(0.1 -0.1)\">\n","      <use xlink:href=\"#DejaVuSans-74\"/>\n","      <use xlink:href=\"#DejaVuSans-65\" x=\"39.208984\"/>\n","      <use xlink:href=\"#DejaVuSans-73\" x=\"100.732422\"/>\n","      <use xlink:href=\"#DejaVuSans-74\" x=\"152.832031\"/>\n","     </g>\n","    </g>\n","   </g>\n","  </g>\n"," </g>\n"," <defs>\n","  <clipPath id=\"paf1508e7f6\">\n","   <rect x=\"51.378125\" y=\"7.2\" width=\"195.3\" height=\"135.9\"/>\n","  </clipPath>\n"," </defs>\n","</svg>\n"],"text/plain":["<Figure size 252x180 with 1 Axes>"]},"metadata":{"needs_background":"light"},"output_type":"display_data"}],"source":["train(lambd=0)"],"id":"-FwnI4bkhFj2"},{"cell_type":"markdown","metadata":{"origin_pos":20,"id":"qa9CKWYWhFj2"},"source":["### [**使用权重衰减**]\n","\n","下面，我们使用权重衰减来运行代码。\n","注意，在这里训练误差增大，但测试误差减小。\n","这正是我们期望从正则化中得到的效果。\n"],"id":"qa9CKWYWhFj2"},{"cell_type":"code","execution_count":null,"metadata":{"execution":{"iopub.execute_input":"2022-07-31T02:22:55.835902Z","iopub.status.busy":"2022-07-31T02:22:55.835673Z","iopub.status.idle":"2022-07-31T02:23:01.979648Z","shell.execute_reply":"2022-07-31T02:23:01.978765Z"},"origin_pos":21,"tab":["pytorch"],"outputId":"a9e1657b-9711-4d71-e957-c4ae96b7a725","id":"6Oby4IXRhFj2"},"outputs":[{"name":"stdout","output_type":"stream","text":["w的L2范数是： 0.3558707535266876\n"]},{"data":{"image/svg+xml":["<?xml version=\"1.0\" encoding=\"utf-8\" standalone=\"no\"?>\n","<!DOCTYPE svg PUBLIC \"-//W3C//DTD SVG 1.1//EN\"\n","  \"http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd\">\n","<svg xmlns:xlink=\"http://www.w3.org/1999/xlink\" width=\"263.421875pt\" height=\"182.600266pt\" viewBox=\"0 0 263.421875 182.600266\" xmlns=\"http://www.w3.org/2000/svg\" version=\"1.1\">\n"," <metadata>\n","  <rdf:RDF xmlns:dc=\"http://purl.org/dc/elements/1.1/\" xmlns:cc=\"http://creativecommons.org/ns#\" xmlns:rdf=\"http://www.w3.org/1999/02/22-rdf-syntax-ns#\">\n","   <cc:Work>\n","    <dc:type rdf:resource=\"http://purl.org/dc/dcmitype/StillImage\"/>\n","    <dc:date>2022-07-31T02:23:01.916702</dc:date>\n","    <dc:format>image/svg+xml</dc:format>\n","    <dc:creator>\n","     <cc:Agent>\n","      <dc:title>Matplotlib v3.5.1, https://matplotlib.org/</dc:title>\n","     </cc:Agent>\n","    </dc:creator>\n","   </cc:Work>\n","  </rdf:RDF>\n"," </metadata>\n"," <defs>\n","  <style type=\"text/css\">*{stroke-linejoin: round; stroke-linecap: butt}</style>\n"," </defs>\n"," <g id=\"figure_1\">\n","  <g id=\"patch_1\">\n","   <path d=\"M 0 182.600266 \n","L 263.421875 182.600266 \n","L 263.421875 -0 \n","L 0 -0 \n","L 0 182.600266 \n","z\n","\" style=\"fill: none\"/>\n","  </g>\n","  <g id=\"axes_1\">\n","   <g id=\"patch_2\">\n","    <path d=\"M 51.378125 145.044016 \n","L 246.678125 145.044016 \n","L 246.678125 9.144016 \n","L 51.378125 9.144016 \n","z\n","\" style=\"fill: #ffffff\"/>\n","   </g>\n","   <g id=\"matplotlib.axis_1\">\n","    <g id=\"xtick_1\">\n","     <g id=\"line2d_1\">\n","      <path d=\"M 82.214967 145.044016 \n","L 82.214967 9.144016 \n","\" clip-path=\"url(#pd83c5e35c7)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n","     </g>\n","     <g id=\"line2d_2\">\n","      <defs>\n","       <path id=\"mc4b0c12c4b\" d=\"M 0 0 \n","L 0 3.5 \n","\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n","      </defs>\n","      <g>\n","       <use xlink:href=\"#mc4b0c12c4b\" x=\"82.214967\" y=\"145.044016\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n","      </g>\n","     </g>\n","     <g id=\"text_1\">\n","      <!-- 20 -->\n","      <g transform=\"translate(75.852467 159.642454)scale(0.1 -0.1)\">\n","       <defs>\n","        <path id=\"DejaVuSans-32\" d=\"M 1228 531 \n","L 3431 531 \n","L 3431 0 \n","L 469 0 \n","L 469 531 \n","Q 828 903 1448 1529 \n","Q 2069 2156 2228 2338 \n","Q 2531 2678 2651 2914 \n","Q 2772 3150 2772 3378 \n","Q 2772 3750 2511 3984 \n","Q 2250 4219 1831 4219 \n","Q 1534 4219 1204 4116 \n","Q 875 4013 500 3803 \n","L 500 4441 \n","Q 881 4594 1212 4672 \n","Q 1544 4750 1819 4750 \n","Q 2544 4750 2975 4387 \n","Q 3406 4025 3406 3419 \n","Q 3406 3131 3298 2873 \n","Q 3191 2616 2906 2266 \n","Q 2828 2175 2409 1742 \n","Q 1991 1309 1228 531 \n","z\n","\" transform=\"scale(0.015625)\"/>\n","        <path id=\"DejaVuSans-30\" d=\"M 2034 4250 \n","Q 1547 4250 1301 3770 \n","Q 1056 3291 1056 2328 \n","Q 1056 1369 1301 889 \n","Q 1547 409 2034 409 \n","Q 2525 409 2770 889 \n","Q 3016 1369 3016 2328 \n","Q 3016 3291 2770 3770 \n","Q 2525 4250 2034 4250 \n","z\n","M 2034 4750 \n","Q 2819 4750 3233 4129 \n","Q 3647 3509 3647 2328 \n","Q 3647 1150 3233 529 \n","Q 2819 -91 2034 -91 \n","Q 1250 -91 836 529 \n","Q 422 1150 422 2328 \n","Q 422 3509 836 4129 \n","Q 1250 4750 2034 4750 \n","z\n","\" transform=\"scale(0.015625)\"/>\n","       </defs>\n","       <use xlink:href=\"#DejaVuSans-32\"/>\n","       <use xlink:href=\"#DejaVuSans-30\" x=\"63.623047\"/>\n","      </g>\n","     </g>\n","    </g>\n","    <g id=\"xtick_2\">\n","     <g id=\"line2d_3\">\n","      <path d=\"M 123.330757 145.044016 \n","L 123.330757 9.144016 \n","\" clip-path=\"url(#pd83c5e35c7)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n","     </g>\n","     <g id=\"line2d_4\">\n","      <g>\n","       <use xlink:href=\"#mc4b0c12c4b\" x=\"123.330757\" y=\"145.044016\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n","      </g>\n","     </g>\n","     <g id=\"text_2\">\n","      <!-- 40 -->\n","      <g transform=\"translate(116.968257 159.642454)scale(0.1 -0.1)\">\n","       <defs>\n","        <path id=\"DejaVuSans-34\" d=\"M 2419 4116 \n","L 825 1625 \n","L 2419 1625 \n","L 2419 4116 \n","z\n","M 2253 4666 \n","L 3047 4666 \n","L 3047 1625 \n","L 3713 1625 \n","L 3713 1100 \n","L 3047 1100 \n","L 3047 0 \n","L 2419 0 \n","L 2419 1100 \n","L 313 1100 \n","L 313 1709 \n","L 2253 4666 \n","z\n","\" transform=\"scale(0.015625)\"/>\n","       </defs>\n","       <use xlink:href=\"#DejaVuSans-34\"/>\n","       <use xlink:href=\"#DejaVuSans-30\" x=\"63.623047\"/>\n","      </g>\n","     </g>\n","    </g>\n","    <g id=\"xtick_3\">\n","     <g id=\"line2d_5\">\n","      <path d=\"M 164.446546 145.044016 \n","L 164.446546 9.144016 \n","\" clip-path=\"url(#pd83c5e35c7)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n","     </g>\n","     <g id=\"line2d_6\">\n","      <g>\n","       <use xlink:href=\"#mc4b0c12c4b\" x=\"164.446546\" y=\"145.044016\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n","      </g>\n","     </g>\n","     <g id=\"text_3\">\n","      <!-- 60 -->\n","      <g transform=\"translate(158.084046 159.642454)scale(0.1 -0.1)\">\n","       <defs>\n","        <path id=\"DejaVuSans-36\" d=\"M 2113 2584 \n","Q 1688 2584 1439 2293 \n","Q 1191 2003 1191 1497 \n","Q 1191 994 1439 701 \n","Q 1688 409 2113 409 \n","Q 2538 409 2786 701 \n","Q 3034 994 3034 1497 \n","Q 3034 2003 2786 2293 \n","Q 2538 2584 2113 2584 \n","z\n","M 3366 4563 \n","L 3366 3988 \n","Q 3128 4100 2886 4159 \n","Q 2644 4219 2406 4219 \n","Q 1781 4219 1451 3797 \n","Q 1122 3375 1075 2522 \n","Q 1259 2794 1537 2939 \n","Q 1816 3084 2150 3084 \n","Q 2853 3084 3261 2657 \n","Q 3669 2231 3669 1497 \n","Q 3669 778 3244 343 \n","Q 2819 -91 2113 -91 \n","Q 1303 -91 875 529 \n","Q 447 1150 447 2328 \n","Q 447 3434 972 4092 \n","Q 1497 4750 2381 4750 \n","Q 2619 4750 2861 4703 \n","Q 3103 4656 3366 4563 \n","z\n","\" transform=\"scale(0.015625)\"/>\n","       </defs>\n","       <use xlink:href=\"#DejaVuSans-36\"/>\n","       <use xlink:href=\"#DejaVuSans-30\" x=\"63.623047\"/>\n","      </g>\n","     </g>\n","    </g>\n","    <g id=\"xtick_4\">\n","     <g id=\"line2d_7\">\n","      <path d=\"M 205.562336 145.044016 \n","L 205.562336 9.144016 \n","\" clip-path=\"url(#pd83c5e35c7)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n","     </g>\n","     <g id=\"line2d_8\">\n","      <g>\n","       <use xlink:href=\"#mc4b0c12c4b\" x=\"205.562336\" y=\"145.044016\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n","      </g>\n","     </g>\n","     <g id=\"text_4\">\n","      <!-- 80 -->\n","      <g transform=\"translate(199.199836 159.642454)scale(0.1 -0.1)\">\n","       <defs>\n","        <path id=\"DejaVuSans-38\" d=\"M 2034 2216 \n","Q 1584 2216 1326 1975 \n","Q 1069 1734 1069 1313 \n","Q 1069 891 1326 650 \n","Q 1584 409 2034 409 \n","Q 2484 409 2743 651 \n","Q 3003 894 3003 1313 \n","Q 3003 1734 2745 1975 \n","Q 2488 2216 2034 2216 \n","z\n","M 1403 2484 \n","Q 997 2584 770 2862 \n","Q 544 3141 544 3541 \n","Q 544 4100 942 4425 \n","Q 1341 4750 2034 4750 \n","Q 2731 4750 3128 4425 \n","Q 3525 4100 3525 3541 \n","Q 3525 3141 3298 2862 \n","Q 3072 2584 2669 2484 \n","Q 3125 2378 3379 2068 \n","Q 3634 1759 3634 1313 \n","Q 3634 634 3220 271 \n","Q 2806 -91 2034 -91 \n","Q 1263 -91 848 271 \n","Q 434 634 434 1313 \n","Q 434 1759 690 2068 \n","Q 947 2378 1403 2484 \n","z\n","M 1172 3481 \n","Q 1172 3119 1398 2916 \n","Q 1625 2713 2034 2713 \n","Q 2441 2713 2670 2916 \n","Q 2900 3119 2900 3481 \n","Q 2900 3844 2670 4047 \n","Q 2441 4250 2034 4250 \n","Q 1625 4250 1398 4047 \n","Q 1172 3844 1172 3481 \n","z\n","\" transform=\"scale(0.015625)\"/>\n","       </defs>\n","       <use xlink:href=\"#DejaVuSans-38\"/>\n","       <use xlink:href=\"#DejaVuSans-30\" x=\"63.623047\"/>\n","      </g>\n","     </g>\n","    </g>\n","    <g id=\"xtick_5\">\n","     <g id=\"line2d_9\">\n","      <path d=\"M 246.678125 145.044016 \n","L 246.678125 9.144016 \n","\" clip-path=\"url(#pd83c5e35c7)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n","     </g>\n","     <g id=\"line2d_10\">\n","      <g>\n","       <use xlink:href=\"#mc4b0c12c4b\" x=\"246.678125\" y=\"145.044016\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n","      </g>\n","     </g>\n","     <g id=\"text_5\">\n","      <!-- 100 -->\n","      <g transform=\"translate(237.134375 159.642454)scale(0.1 -0.1)\">\n","       <defs>\n","        <path id=\"DejaVuSans-31\" d=\"M 794 531 \n","L 1825 531 \n","L 1825 4091 \n","L 703 3866 \n","L 703 4441 \n","L 1819 4666 \n","L 2450 4666 \n","L 2450 531 \n","L 3481 531 \n","L 3481 0 \n","L 794 0 \n","L 794 531 \n","z\n","\" transform=\"scale(0.015625)\"/>\n","       </defs>\n","       <use xlink:href=\"#DejaVuSans-31\"/>\n","       <use xlink:href=\"#DejaVuSans-30\" x=\"63.623047\"/>\n","       <use xlink:href=\"#DejaVuSans-30\" x=\"127.246094\"/>\n","      </g>\n","     </g>\n","    </g>\n","    <g id=\"text_6\">\n","     <!-- epochs -->\n","     <g transform=\"translate(131.195312 173.320579)scale(0.1 -0.1)\">\n","      <defs>\n","       <path id=\"DejaVuSans-65\" d=\"M 3597 1894 \n","L 3597 1613 \n","L 953 1613 \n","Q 991 1019 1311 708 \n","Q 1631 397 2203 397 \n","Q 2534 397 2845 478 \n","Q 3156 559 3463 722 \n","L 3463 178 \n","Q 3153 47 2828 -22 \n","Q 2503 -91 2169 -91 \n","Q 1331 -91 842 396 \n","Q 353 884 353 1716 \n","Q 353 2575 817 3079 \n","Q 1281 3584 2069 3584 \n","Q 2775 3584 3186 3129 \n","Q 3597 2675 3597 1894 \n","z\n","M 3022 2063 \n","Q 3016 2534 2758 2815 \n","Q 2500 3097 2075 3097 \n","Q 1594 3097 1305 2825 \n","Q 1016 2553 972 2059 \n","L 3022 2063 \n","z\n","\" transform=\"scale(0.015625)\"/>\n","       <path id=\"DejaVuSans-70\" d=\"M 1159 525 \n","L 1159 -1331 \n","L 581 -1331 \n","L 581 3500 \n","L 1159 3500 \n","L 1159 2969 \n","Q 1341 3281 1617 3432 \n","Q 1894 3584 2278 3584 \n","Q 2916 3584 3314 3078 \n","Q 3713 2572 3713 1747 \n","Q 3713 922 3314 415 \n","Q 2916 -91 2278 -91 \n","Q 1894 -91 1617 61 \n","Q 1341 213 1159 525 \n","z\n","M 3116 1747 \n","Q 3116 2381 2855 2742 \n","Q 2594 3103 2138 3103 \n","Q 1681 3103 1420 2742 \n","Q 1159 2381 1159 1747 \n","Q 1159 1113 1420 752 \n","Q 1681 391 2138 391 \n","Q 2594 391 2855 752 \n","Q 3116 1113 3116 1747 \n","z\n","\" transform=\"scale(0.015625)\"/>\n","       <path id=\"DejaVuSans-6f\" d=\"M 1959 3097 \n","Q 1497 3097 1228 2736 \n","Q 959 2375 959 1747 \n","Q 959 1119 1226 758 \n","Q 1494 397 1959 397 \n","Q 2419 397 2687 759 \n","Q 2956 1122 2956 1747 \n","Q 2956 2369 2687 2733 \n","Q 2419 3097 1959 3097 \n","z\n","M 1959 3584 \n","Q 2709 3584 3137 3096 \n","Q 3566 2609 3566 1747 \n","Q 3566 888 3137 398 \n","Q 2709 -91 1959 -91 \n","Q 1206 -91 779 398 \n","Q 353 888 353 1747 \n","Q 353 2609 779 3096 \n","Q 1206 3584 1959 3584 \n","z\n","\" transform=\"scale(0.015625)\"/>\n","       <path id=\"DejaVuSans-63\" d=\"M 3122 3366 \n","L 3122 2828 \n","Q 2878 2963 2633 3030 \n","Q 2388 3097 2138 3097 \n","Q 1578 3097 1268 2742 \n","Q 959 2388 959 1747 \n","Q 959 1106 1268 751 \n","Q 1578 397 2138 397 \n","Q 2388 397 2633 464 \n","Q 2878 531 3122 666 \n","L 3122 134 \n","Q 2881 22 2623 -34 \n","Q 2366 -91 2075 -91 \n","Q 1284 -91 818 406 \n","Q 353 903 353 1747 \n","Q 353 2603 823 3093 \n","Q 1294 3584 2113 3584 \n","Q 2378 3584 2631 3529 \n","Q 2884 3475 3122 3366 \n","z\n","\" transform=\"scale(0.015625)\"/>\n","       <path id=\"DejaVuSans-68\" d=\"M 3513 2113 \n","L 3513 0 \n","L 2938 0 \n","L 2938 2094 \n","Q 2938 2591 2744 2837 \n","Q 2550 3084 2163 3084 \n","Q 1697 3084 1428 2787 \n","Q 1159 2491 1159 1978 \n","L 1159 0 \n","L 581 0 \n","L 581 4863 \n","L 1159 4863 \n","L 1159 2956 \n","Q 1366 3272 1645 3428 \n","Q 1925 3584 2291 3584 \n","Q 2894 3584 3203 3211 \n","Q 3513 2838 3513 2113 \n","z\n","\" transform=\"scale(0.015625)\"/>\n","       <path id=\"DejaVuSans-73\" d=\"M 2834 3397 \n","L 2834 2853 \n","Q 2591 2978 2328 3040 \n","Q 2066 3103 1784 3103 \n","Q 1356 3103 1142 2972 \n","Q 928 2841 928 2578 \n","Q 928 2378 1081 2264 \n","Q 1234 2150 1697 2047 \n","L 1894 2003 \n","Q 2506 1872 2764 1633 \n","Q 3022 1394 3022 966 \n","Q 3022 478 2636 193 \n","Q 2250 -91 1575 -91 \n","Q 1294 -91 989 -36 \n","Q 684 19 347 128 \n","L 347 722 \n","Q 666 556 975 473 \n","Q 1284 391 1588 391 \n","Q 1994 391 2212 530 \n","Q 2431 669 2431 922 \n","Q 2431 1156 2273 1281 \n","Q 2116 1406 1581 1522 \n","L 1381 1569 \n","Q 847 1681 609 1914 \n","Q 372 2147 372 2553 \n","Q 372 3047 722 3315 \n","Q 1072 3584 1716 3584 \n","Q 2034 3584 2315 3537 \n","Q 2597 3491 2834 3397 \n","z\n","\" transform=\"scale(0.015625)\"/>\n","      </defs>\n","      <use xlink:href=\"#DejaVuSans-65\"/>\n","      <use xlink:href=\"#DejaVuSans-70\" x=\"61.523438\"/>\n","      <use xlink:href=\"#DejaVuSans-6f\" x=\"125\"/>\n","      <use xlink:href=\"#DejaVuSans-63\" x=\"186.181641\"/>\n","      <use xlink:href=\"#DejaVuSans-68\" x=\"241.162109\"/>\n","      <use xlink:href=\"#DejaVuSans-73\" x=\"304.541016\"/>\n","     </g>\n","    </g>\n","   </g>\n","   <g id=\"matplotlib.axis_2\">\n","    <g id=\"ytick_1\">\n","     <g id=\"line2d_11\">\n","      <path d=\"M 51.378125 112.581014 \n","L 246.678125 112.581014 \n","\" clip-path=\"url(#pd83c5e35c7)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n","     </g>\n","     <g id=\"line2d_12\">\n","      <defs>\n","       <path id=\"ma8aaa4f60f\" d=\"M 0 0 \n","L -3.5 0 \n","\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n","      </defs>\n","      <g>\n","       <use xlink:href=\"#ma8aaa4f60f\" x=\"51.378125\" y=\"112.581014\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n","      </g>\n","     </g>\n","     <g id=\"text_7\">\n","      <!-- $\\mathdefault{10^{-2}}$ -->\n","      <g transform=\"translate(20.878125 116.380233)scale(0.1 -0.1)\">\n","       <defs>\n","        <path id=\"DejaVuSans-2212\" d=\"M 678 2272 \n","L 4684 2272 \n","L 4684 1741 \n","L 678 1741 \n","L 678 2272 \n","z\n","\" transform=\"scale(0.015625)\"/>\n","       </defs>\n","       <use xlink:href=\"#DejaVuSans-31\" transform=\"translate(0 0.765625)\"/>\n","       <use xlink:href=\"#DejaVuSans-30\" transform=\"translate(63.623047 0.765625)\"/>\n","       <use xlink:href=\"#DejaVuSans-2212\" transform=\"translate(128.203125 39.046875)scale(0.7)\"/>\n","       <use xlink:href=\"#DejaVuSans-32\" transform=\"translate(186.855469 39.046875)scale(0.7)\"/>\n","      </g>\n","     </g>\n","    </g>\n","    <g id=\"ytick_2\">\n","     <g id=\"line2d_13\">\n","      <path d=\"M 51.378125 61.790117 \n","L 246.678125 61.790117 \n","\" clip-path=\"url(#pd83c5e35c7)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n","     </g>\n","     <g id=\"line2d_14\">\n","      <g>\n","       <use xlink:href=\"#ma8aaa4f60f\" x=\"51.378125\" y=\"61.790117\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n","      </g>\n","     </g>\n","     <g id=\"text_8\">\n","      <!-- $\\mathdefault{10^{0}}$ -->\n","      <g transform=\"translate(26.778125 65.589335)scale(0.1 -0.1)\">\n","       <use xlink:href=\"#DejaVuSans-31\" transform=\"translate(0 0.765625)\"/>\n","       <use xlink:href=\"#DejaVuSans-30\" transform=\"translate(63.623047 0.765625)\"/>\n","       <use xlink:href=\"#DejaVuSans-30\" transform=\"translate(128.203125 39.046875)scale(0.7)\"/>\n","      </g>\n","     </g>\n","    </g>\n","    <g id=\"ytick_3\">\n","     <g id=\"line2d_15\">\n","      <path d=\"M 51.378125 10.999219 \n","L 246.678125 10.999219 \n","\" clip-path=\"url(#pd83c5e35c7)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n","     </g>\n","     <g id=\"line2d_16\">\n","      <g>\n","       <use xlink:href=\"#ma8aaa4f60f\" x=\"51.378125\" y=\"10.999219\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n","      </g>\n","     </g>\n","     <g id=\"text_9\">\n","      <!-- $\\mathdefault{10^{2}}$ -->\n","      <g transform=\"translate(26.778125 14.798437)scale(0.1 -0.1)\">\n","       <use xlink:href=\"#DejaVuSans-31\" transform=\"translate(0 0.765625)\"/>\n","       <use xlink:href=\"#DejaVuSans-30\" transform=\"translate(63.623047 0.765625)\"/>\n","       <use xlink:href=\"#DejaVuSans-32\" transform=\"translate(128.203125 39.046875)scale(0.7)\"/>\n","      </g>\n","     </g>\n","    </g>\n","    <g id=\"text_10\">\n","     <!-- loss -->\n","     <g transform=\"translate(14.798437 86.751829)rotate(-90)scale(0.1 -0.1)\">\n","      <defs>\n","       <path id=\"DejaVuSans-6c\" d=\"M 603 4863 \n","L 1178 4863 \n","L 1178 0 \n","L 603 0 \n","L 603 4863 \n","z\n","\" transform=\"scale(0.015625)\"/>\n","      </defs>\n","      <use xlink:href=\"#DejaVuSans-6c\"/>\n","      <use xlink:href=\"#DejaVuSans-6f\" x=\"27.783203\"/>\n","      <use xlink:href=\"#DejaVuSans-73\" x=\"88.964844\"/>\n","      <use xlink:href=\"#DejaVuSans-73\" x=\"141.064453\"/>\n","     </g>\n","    </g>\n","   </g>\n","   <g id=\"line2d_17\">\n","    <path d=\"M 51.378125 34.334134 \n","L 61.657072 50.369663 \n","L 71.93602 65.62548 \n","L 82.214967 80.348757 \n","L 92.493914 94.697254 \n","L 102.772862 108.615806 \n","L 113.051809 121.476602 \n","L 123.330757 131.489245 \n","L 133.609704 136.728038 \n","L 143.888651 138.225215 \n","L 154.167599 138.387269 \n","L 164.446546 138.194005 \n","L 174.725493 138.189174 \n","L 185.004441 138.179419 \n","L 195.283388 138.096809 \n","L 205.562336 138.364611 \n","L 215.841283 138.475723 \n","L 226.12023 138.682048 \n","L 236.399178 138.846207 \n","L 246.678125 138.866744 \n","\" clip-path=\"url(#pd83c5e35c7)\" style=\"fill: none; stroke: #1f77b4; stroke-width: 1.5; stroke-linecap: square\"/>\n","   </g>\n","   <g id=\"line2d_18\">\n","    <path d=\"M 51.378125 15.321289 \n","L 61.657072 19.257826 \n","L 71.93602 23.181495 \n","L 82.214967 27.1216 \n","L 92.493914 31.078954 \n","L 102.772862 35.050837 \n","L 113.051809 39.032385 \n","L 123.330757 43.02071 \n","L 133.609704 47.012623 \n","L 143.888651 51.005697 \n","L 154.167599 54.99629 \n","L 164.446546 58.980406 \n","L 174.725493 62.952397 \n","L 185.004441 66.90531 \n","L 195.283388 70.827173 \n","L 205.562336 74.703102 \n","L 215.841283 78.512446 \n","L 226.12023 82.224176 \n","L 236.399178 85.803689 \n","L 246.678125 89.204236 \n","\" clip-path=\"url(#pd83c5e35c7)\" style=\"fill: none; stroke-dasharray: 5.55,2.4; stroke-dashoffset: 0; stroke: #bf00bf; stroke-width: 1.5\"/>\n","   </g>\n","   <g id=\"patch_3\">\n","    <path d=\"M 51.378125 145.044016 \n","L 51.378125 9.144016 \n","\" style=\"fill: none; stroke: #000000; stroke-width: 0.8; stroke-linejoin: miter; stroke-linecap: square\"/>\n","   </g>\n","   <g id=\"patch_4\">\n","    <path d=\"M 246.678125 145.044016 \n","L 246.678125 9.144016 \n","\" style=\"fill: none; stroke: #000000; stroke-width: 0.8; stroke-linejoin: miter; stroke-linecap: square\"/>\n","   </g>\n","   <g id=\"patch_5\">\n","    <path d=\"M 51.378125 145.044016 \n","L 246.678125 145.044016 \n","\" style=\"fill: none; stroke: #000000; stroke-width: 0.8; stroke-linejoin: miter; stroke-linecap: square\"/>\n","   </g>\n","   <g id=\"patch_6\">\n","    <path d=\"M 51.378125 9.144016 \n","L 246.678125 9.144016 \n","\" style=\"fill: none; stroke: #000000; stroke-width: 0.8; stroke-linejoin: miter; stroke-linecap: square\"/>\n","   </g>\n","   <g id=\"legend_1\">\n","    <g id=\"patch_7\">\n","     <path d=\"M 184.403125 46.500266 \n","L 239.678125 46.500266 \n","Q 241.678125 46.500266 241.678125 44.500266 \n","L 241.678125 16.144016 \n","Q 241.678125 14.144016 239.678125 14.144016 \n","L 184.403125 14.144016 \n","Q 182.403125 14.144016 182.403125 16.144016 \n","L 182.403125 44.500266 \n","Q 182.403125 46.500266 184.403125 46.500266 \n","z\n","\" style=\"fill: #ffffff; opacity: 0.8; stroke: #cccccc; stroke-linejoin: miter\"/>\n","    </g>\n","    <g id=\"line2d_19\">\n","     <path d=\"M 186.403125 22.242454 \n","L 196.403125 22.242454 \n","L 206.403125 22.242454 \n","\" style=\"fill: none; stroke: #1f77b4; stroke-width: 1.5; stroke-linecap: square\"/>\n","    </g>\n","    <g id=\"text_11\">\n","     <!-- train -->\n","     <g transform=\"translate(214.403125 25.742454)scale(0.1 -0.1)\">\n","      <defs>\n","       <path id=\"DejaVuSans-74\" d=\"M 1172 4494 \n","L 1172 3500 \n","L 2356 3500 \n","L 2356 3053 \n","L 1172 3053 \n","L 1172 1153 \n","Q 1172 725 1289 603 \n","Q 1406 481 1766 481 \n","L 2356 481 \n","L 2356 0 \n","L 1766 0 \n","Q 1100 0 847 248 \n","Q 594 497 594 1153 \n","L 594 3053 \n","L 172 3053 \n","L 172 3500 \n","L 594 3500 \n","L 594 4494 \n","L 1172 4494 \n","z\n","\" transform=\"scale(0.015625)\"/>\n","       <path id=\"DejaVuSans-72\" d=\"M 2631 2963 \n","Q 2534 3019 2420 3045 \n","Q 2306 3072 2169 3072 \n","Q 1681 3072 1420 2755 \n","Q 1159 2438 1159 1844 \n","L 1159 0 \n","L 581 0 \n","L 581 3500 \n","L 1159 3500 \n","L 1159 2956 \n","Q 1341 3275 1631 3429 \n","Q 1922 3584 2338 3584 \n","Q 2397 3584 2469 3576 \n","Q 2541 3569 2628 3553 \n","L 2631 2963 \n","z\n","\" transform=\"scale(0.015625)\"/>\n","       <path id=\"DejaVuSans-61\" d=\"M 2194 1759 \n","Q 1497 1759 1228 1600 \n","Q 959 1441 959 1056 \n","Q 959 750 1161 570 \n","Q 1363 391 1709 391 \n","Q 2188 391 2477 730 \n","Q 2766 1069 2766 1631 \n","L 2766 1759 \n","L 2194 1759 \n","z\n","M 3341 1997 \n","L 3341 0 \n","L 2766 0 \n","L 2766 531 \n","Q 2569 213 2275 61 \n","Q 1981 -91 1556 -91 \n","Q 1019 -91 701 211 \n","Q 384 513 384 1019 \n","Q 384 1609 779 1909 \n","Q 1175 2209 1959 2209 \n","L 2766 2209 \n","L 2766 2266 \n","Q 2766 2663 2505 2880 \n","Q 2244 3097 1772 3097 \n","Q 1472 3097 1187 3025 \n","Q 903 2953 641 2809 \n","L 641 3341 \n","Q 956 3463 1253 3523 \n","Q 1550 3584 1831 3584 \n","Q 2591 3584 2966 3190 \n","Q 3341 2797 3341 1997 \n","z\n","\" transform=\"scale(0.015625)\"/>\n","       <path id=\"DejaVuSans-69\" d=\"M 603 3500 \n","L 1178 3500 \n","L 1178 0 \n","L 603 0 \n","L 603 3500 \n","z\n","M 603 4863 \n","L 1178 4863 \n","L 1178 4134 \n","L 603 4134 \n","L 603 4863 \n","z\n","\" transform=\"scale(0.015625)\"/>\n","       <path id=\"DejaVuSans-6e\" d=\"M 3513 2113 \n","L 3513 0 \n","L 2938 0 \n","L 2938 2094 \n","Q 2938 2591 2744 2837 \n","Q 2550 3084 2163 3084 \n","Q 1697 3084 1428 2787 \n","Q 1159 2491 1159 1978 \n","L 1159 0 \n","L 581 0 \n","L 581 3500 \n","L 1159 3500 \n","L 1159 2956 \n","Q 1366 3272 1645 3428 \n","Q 1925 3584 2291 3584 \n","Q 2894 3584 3203 3211 \n","Q 3513 2838 3513 2113 \n","z\n","\" transform=\"scale(0.015625)\"/>\n","      </defs>\n","      <use xlink:href=\"#DejaVuSans-74\"/>\n","      <use xlink:href=\"#DejaVuSans-72\" x=\"39.208984\"/>\n","      <use xlink:href=\"#DejaVuSans-61\" x=\"80.322266\"/>\n","      <use xlink:href=\"#DejaVuSans-69\" x=\"141.601562\"/>\n","      <use xlink:href=\"#DejaVuSans-6e\" x=\"169.384766\"/>\n","     </g>\n","    </g>\n","    <g id=\"line2d_20\">\n","     <path d=\"M 186.403125 36.920579 \n","L 196.403125 36.920579 \n","L 206.403125 36.920579 \n","\" style=\"fill: none; stroke-dasharray: 5.55,2.4; stroke-dashoffset: 0; stroke: #bf00bf; stroke-width: 1.5\"/>\n","    </g>\n","    <g id=\"text_12\">\n","     <!-- test -->\n","     <g transform=\"translate(214.403125 40.420579)scale(0.1 -0.1)\">\n","      <use xlink:href=\"#DejaVuSans-74\"/>\n","      <use xlink:href=\"#DejaVuSans-65\" x=\"39.208984\"/>\n","      <use xlink:href=\"#DejaVuSans-73\" x=\"100.732422\"/>\n","      <use xlink:href=\"#DejaVuSans-74\" x=\"152.832031\"/>\n","     </g>\n","    </g>\n","   </g>\n","  </g>\n"," </g>\n"," <defs>\n","  <clipPath id=\"pd83c5e35c7\">\n","   <rect x=\"51.378125\" y=\"9.144016\" width=\"195.3\" height=\"135.9\"/>\n","  </clipPath>\n"," </defs>\n","</svg>\n"],"text/plain":["<Figure size 252x180 with 1 Axes>"]},"metadata":{"needs_background":"light"},"output_type":"display_data"}],"source":["train(lambd=3)"],"id":"6Oby4IXRhFj2"},{"cell_type":"markdown","metadata":{"origin_pos":22,"id":"-q6QdmwFhFj2"},"source":["## [**简洁实现**]\n","\n","由于权重衰减在神经网络优化中很常用，\n","深度学习框架为了便于我们使用权重衰减，\n","将权重衰减集成到优化算法中，以便与任何损失函数结合使用。\n","此外，这种集成还有计算上的好处，\n","允许在不增加任何额外的计算开销的情况下向算法中添加权重衰减。\n","由于更新的权重衰减部分仅依赖于每个参数的当前值，\n","因此优化器必须至少接触每个参数一次。\n"],"id":"-q6QdmwFhFj2"},{"cell_type":"markdown","metadata":{"origin_pos":24,"tab":["pytorch"],"id":"8SwZovPahFj2"},"source":["在下面的代码中，我们在实例化优化器时直接通过`weight_decay`指定weight decay超参数。\n","默认情况下，PyTorch同时衰减权重和偏移。\n","这里我们只为权重设置了`weight_decay`，所以偏置参数$b$不会衰减。\n"],"id":"8SwZovPahFj2"},{"cell_type":"code","execution_count":null,"metadata":{"execution":{"iopub.execute_input":"2022-07-31T02:23:01.983489Z","iopub.status.busy":"2022-07-31T02:23:01.982884Z","iopub.status.idle":"2022-07-31T02:23:01.991713Z","shell.execute_reply":"2022-07-31T02:23:01.990937Z"},"origin_pos":27,"tab":["pytorch"],"id":"Y2FtRVwehFj2"},"outputs":[],"source":["def train_concise(wd):\n","    net = nn.Sequential(nn.Linear(num_inputs, 1))\n","    for param in net.parameters():\n","        param.data.normal_()\n","    loss = nn.MSELoss(reduction='none')\n","    num_epochs, lr = 100, 0.003\n","    # 偏置参数没有衰减\n","    trainer = torch.optim.SGD([\n","        {\"params\":net[0].weight,'weight_decay': wd},\n","        {\"params\":net[0].bias}], lr=lr)\n","    animator = d2l.Animator(xlabel='epochs', ylabel='loss', yscale='log',\n","                            xlim=[5, num_epochs], legend=['train', 'test'])\n","    for epoch in range(num_epochs):\n","        for X, y in train_iter:\n","            trainer.zero_grad()\n","            l = loss(net(X), y)\n","            l.mean().backward()\n","            trainer.step()\n","        if (epoch + 1) % 5 == 0:\n","            animator.add(epoch + 1,\n","                         (d2l.evaluate_loss(net, train_iter, loss),\n","                          d2l.evaluate_loss(net, test_iter, loss)))\n","    print('w的L2范数：', net[0].weight.norm().item())"],"id":"Y2FtRVwehFj2"},{"cell_type":"markdown","metadata":{"origin_pos":29,"id":"Gx2dnVwbhFj2"},"source":["[**这些图看起来和我们从零开始实现权重衰减时的图相同**]。\n","然而，它们运行得更快，更容易实现。\n","对于更复杂的问题，这一好处将变得更加明显。\n"],"id":"Gx2dnVwbhFj2"},{"cell_type":"code","execution_count":null,"metadata":{"execution":{"iopub.execute_input":"2022-07-31T02:23:01.995096Z","iopub.status.busy":"2022-07-31T02:23:01.994625Z","iopub.status.idle":"2022-07-31T02:23:07.965364Z","shell.execute_reply":"2022-07-31T02:23:07.964719Z"},"origin_pos":30,"tab":["pytorch"],"outputId":"20639d14-5066-4be4-ba3e-3518fc97ee92","id":"7vqPr0M3hFj2"},"outputs":[{"name":"stdout","output_type":"stream","text":["w的L2范数： 14.026016235351562\n"]},{"data":{"image/svg+xml":["<?xml version=\"1.0\" encoding=\"utf-8\" standalone=\"no\"?>\n","<!DOCTYPE svg PUBLIC \"-//W3C//DTD SVG 1.1//EN\"\n","  \"http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd\">\n","<svg xmlns:xlink=\"http://www.w3.org/1999/xlink\" width=\"263.421875pt\" height=\"180.65625pt\" viewBox=\"0 0 263.421875 180.65625\" xmlns=\"http://www.w3.org/2000/svg\" version=\"1.1\">\n"," <metadata>\n","  <rdf:RDF xmlns:dc=\"http://purl.org/dc/elements/1.1/\" xmlns:cc=\"http://creativecommons.org/ns#\" xmlns:rdf=\"http://www.w3.org/1999/02/22-rdf-syntax-ns#\">\n","   <cc:Work>\n","    <dc:type rdf:resource=\"http://purl.org/dc/dcmitype/StillImage\"/>\n","    <dc:date>2022-07-31T02:23:07.884250</dc:date>\n","    <dc:format>image/svg+xml</dc:format>\n","    <dc:creator>\n","     <cc:Agent>\n","      <dc:title>Matplotlib v3.5.1, https://matplotlib.org/</dc:title>\n","     </cc:Agent>\n","    </dc:creator>\n","   </cc:Work>\n","  </rdf:RDF>\n"," </metadata>\n"," <defs>\n","  <style type=\"text/css\">*{stroke-linejoin: round; stroke-linecap: butt}</style>\n"," </defs>\n"," <g id=\"figure_1\">\n","  <g id=\"patch_1\">\n","   <path d=\"M 0 180.65625 \n","L 263.421875 180.65625 \n","L 263.421875 0 \n","L 0 0 \n","L 0 180.65625 \n","z\n","\" style=\"fill: none\"/>\n","  </g>\n","  <g id=\"axes_1\">\n","   <g id=\"patch_2\">\n","    <path d=\"M 51.378125 143.1 \n","L 246.678125 143.1 \n","L 246.678125 7.2 \n","L 51.378125 7.2 \n","z\n","\" style=\"fill: #ffffff\"/>\n","   </g>\n","   <g id=\"matplotlib.axis_1\">\n","    <g id=\"xtick_1\">\n","     <g id=\"line2d_1\">\n","      <path d=\"M 82.214967 143.1 \n","L 82.214967 7.2 \n","\" clip-path=\"url(#p5ed042e645)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n","     </g>\n","     <g id=\"line2d_2\">\n","      <defs>\n","       <path id=\"me20a397f43\" d=\"M 0 0 \n","L 0 3.5 \n","\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n","      </defs>\n","      <g>\n","       <use xlink:href=\"#me20a397f43\" x=\"82.214967\" y=\"143.1\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n","      </g>\n","     </g>\n","     <g id=\"text_1\">\n","      <!-- 20 -->\n","      <g transform=\"translate(75.852467 157.698438)scale(0.1 -0.1)\">\n","       <defs>\n","        <path id=\"DejaVuSans-32\" d=\"M 1228 531 \n","L 3431 531 \n","L 3431 0 \n","L 469 0 \n","L 469 531 \n","Q 828 903 1448 1529 \n","Q 2069 2156 2228 2338 \n","Q 2531 2678 2651 2914 \n","Q 2772 3150 2772 3378 \n","Q 2772 3750 2511 3984 \n","Q 2250 4219 1831 4219 \n","Q 1534 4219 1204 4116 \n","Q 875 4013 500 3803 \n","L 500 4441 \n","Q 881 4594 1212 4672 \n","Q 1544 4750 1819 4750 \n","Q 2544 4750 2975 4387 \n","Q 3406 4025 3406 3419 \n","Q 3406 3131 3298 2873 \n","Q 3191 2616 2906 2266 \n","Q 2828 2175 2409 1742 \n","Q 1991 1309 1228 531 \n","z\n","\" transform=\"scale(0.015625)\"/>\n","        <path id=\"DejaVuSans-30\" d=\"M 2034 4250 \n","Q 1547 4250 1301 3770 \n","Q 1056 3291 1056 2328 \n","Q 1056 1369 1301 889 \n","Q 1547 409 2034 409 \n","Q 2525 409 2770 889 \n","Q 3016 1369 3016 2328 \n","Q 3016 3291 2770 3770 \n","Q 2525 4250 2034 4250 \n","z\n","M 2034 4750 \n","Q 2819 4750 3233 4129 \n","Q 3647 3509 3647 2328 \n","Q 3647 1150 3233 529 \n","Q 2819 -91 2034 -91 \n","Q 1250 -91 836 529 \n","Q 422 1150 422 2328 \n","Q 422 3509 836 4129 \n","Q 1250 4750 2034 4750 \n","z\n","\" transform=\"scale(0.015625)\"/>\n","       </defs>\n","       <use xlink:href=\"#DejaVuSans-32\"/>\n","       <use xlink:href=\"#DejaVuSans-30\" x=\"63.623047\"/>\n","      </g>\n","     </g>\n","    </g>\n","    <g id=\"xtick_2\">\n","     <g id=\"line2d_3\">\n","      <path d=\"M 123.330757 143.1 \n","L 123.330757 7.2 \n","\" clip-path=\"url(#p5ed042e645)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n","     </g>\n","     <g id=\"line2d_4\">\n","      <g>\n","       <use xlink:href=\"#me20a397f43\" x=\"123.330757\" y=\"143.1\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n","      </g>\n","     </g>\n","     <g id=\"text_2\">\n","      <!-- 40 -->\n","      <g transform=\"translate(116.968257 157.698438)scale(0.1 -0.1)\">\n","       <defs>\n","        <path id=\"DejaVuSans-34\" d=\"M 2419 4116 \n","L 825 1625 \n","L 2419 1625 \n","L 2419 4116 \n","z\n","M 2253 4666 \n","L 3047 4666 \n","L 3047 1625 \n","L 3713 1625 \n","L 3713 1100 \n","L 3047 1100 \n","L 3047 0 \n","L 2419 0 \n","L 2419 1100 \n","L 313 1100 \n","L 313 1709 \n","L 2253 4666 \n","z\n","\" transform=\"scale(0.015625)\"/>\n","       </defs>\n","       <use xlink:href=\"#DejaVuSans-34\"/>\n","       <use xlink:href=\"#DejaVuSans-30\" x=\"63.623047\"/>\n","      </g>\n","     </g>\n","    </g>\n","    <g id=\"xtick_3\">\n","     <g id=\"line2d_5\">\n","      <path d=\"M 164.446546 143.1 \n","L 164.446546 7.2 \n","\" clip-path=\"url(#p5ed042e645)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n","     </g>\n","     <g id=\"line2d_6\">\n","      <g>\n","       <use xlink:href=\"#me20a397f43\" x=\"164.446546\" y=\"143.1\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n","      </g>\n","     </g>\n","     <g id=\"text_3\">\n","      <!-- 60 -->\n","      <g transform=\"translate(158.084046 157.698438)scale(0.1 -0.1)\">\n","       <defs>\n","        <path id=\"DejaVuSans-36\" d=\"M 2113 2584 \n","Q 1688 2584 1439 2293 \n","Q 1191 2003 1191 1497 \n","Q 1191 994 1439 701 \n","Q 1688 409 2113 409 \n","Q 2538 409 2786 701 \n","Q 3034 994 3034 1497 \n","Q 3034 2003 2786 2293 \n","Q 2538 2584 2113 2584 \n","z\n","M 3366 4563 \n","L 3366 3988 \n","Q 3128 4100 2886 4159 \n","Q 2644 4219 2406 4219 \n","Q 1781 4219 1451 3797 \n","Q 1122 3375 1075 2522 \n","Q 1259 2794 1537 2939 \n","Q 1816 3084 2150 3084 \n","Q 2853 3084 3261 2657 \n","Q 3669 2231 3669 1497 \n","Q 3669 778 3244 343 \n","Q 2819 -91 2113 -91 \n","Q 1303 -91 875 529 \n","Q 447 1150 447 2328 \n","Q 447 3434 972 4092 \n","Q 1497 4750 2381 4750 \n","Q 2619 4750 2861 4703 \n","Q 3103 4656 3366 4563 \n","z\n","\" transform=\"scale(0.015625)\"/>\n","       </defs>\n","       <use xlink:href=\"#DejaVuSans-36\"/>\n","       <use xlink:href=\"#DejaVuSans-30\" x=\"63.623047\"/>\n","      </g>\n","     </g>\n","    </g>\n","    <g id=\"xtick_4\">\n","     <g id=\"line2d_7\">\n","      <path d=\"M 205.562336 143.1 \n","L 205.562336 7.2 \n","\" clip-path=\"url(#p5ed042e645)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n","     </g>\n","     <g id=\"line2d_8\">\n","      <g>\n","       <use xlink:href=\"#me20a397f43\" x=\"205.562336\" y=\"143.1\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n","      </g>\n","     </g>\n","     <g id=\"text_4\">\n","      <!-- 80 -->\n","      <g transform=\"translate(199.199836 157.698438)scale(0.1 -0.1)\">\n","       <defs>\n","        <path id=\"DejaVuSans-38\" d=\"M 2034 2216 \n","Q 1584 2216 1326 1975 \n","Q 1069 1734 1069 1313 \n","Q 1069 891 1326 650 \n","Q 1584 409 2034 409 \n","Q 2484 409 2743 651 \n","Q 3003 894 3003 1313 \n","Q 3003 1734 2745 1975 \n","Q 2488 2216 2034 2216 \n","z\n","M 1403 2484 \n","Q 997 2584 770 2862 \n","Q 544 3141 544 3541 \n","Q 544 4100 942 4425 \n","Q 1341 4750 2034 4750 \n","Q 2731 4750 3128 4425 \n","Q 3525 4100 3525 3541 \n","Q 3525 3141 3298 2862 \n","Q 3072 2584 2669 2484 \n","Q 3125 2378 3379 2068 \n","Q 3634 1759 3634 1313 \n","Q 3634 634 3220 271 \n","Q 2806 -91 2034 -91 \n","Q 1263 -91 848 271 \n","Q 434 634 434 1313 \n","Q 434 1759 690 2068 \n","Q 947 2378 1403 2484 \n","z\n","M 1172 3481 \n","Q 1172 3119 1398 2916 \n","Q 1625 2713 2034 2713 \n","Q 2441 2713 2670 2916 \n","Q 2900 3119 2900 3481 \n","Q 2900 3844 2670 4047 \n","Q 2441 4250 2034 4250 \n","Q 1625 4250 1398 4047 \n","Q 1172 3844 1172 3481 \n","z\n","\" transform=\"scale(0.015625)\"/>\n","       </defs>\n","       <use xlink:href=\"#DejaVuSans-38\"/>\n","       <use xlink:href=\"#DejaVuSans-30\" x=\"63.623047\"/>\n","      </g>\n","     </g>\n","    </g>\n","    <g id=\"xtick_5\">\n","     <g id=\"line2d_9\">\n","      <path d=\"M 246.678125 143.1 \n","L 246.678125 7.2 \n","\" clip-path=\"url(#p5ed042e645)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n","     </g>\n","     <g id=\"line2d_10\">\n","      <g>\n","       <use xlink:href=\"#me20a397f43\" x=\"246.678125\" y=\"143.1\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n","      </g>\n","     </g>\n","     <g id=\"text_5\">\n","      <!-- 100 -->\n","      <g transform=\"translate(237.134375 157.698438)scale(0.1 -0.1)\">\n","       <defs>\n","        <path id=\"DejaVuSans-31\" d=\"M 794 531 \n","L 1825 531 \n","L 1825 4091 \n","L 703 3866 \n","L 703 4441 \n","L 1819 4666 \n","L 2450 4666 \n","L 2450 531 \n","L 3481 531 \n","L 3481 0 \n","L 794 0 \n","L 794 531 \n","z\n","\" transform=\"scale(0.015625)\"/>\n","       </defs>\n","       <use xlink:href=\"#DejaVuSans-31\"/>\n","       <use xlink:href=\"#DejaVuSans-30\" x=\"63.623047\"/>\n","       <use xlink:href=\"#DejaVuSans-30\" x=\"127.246094\"/>\n","      </g>\n","     </g>\n","    </g>\n","    <g id=\"text_6\">\n","     <!-- epochs -->\n","     <g transform=\"translate(131.195312 171.376563)scale(0.1 -0.1)\">\n","      <defs>\n","       <path id=\"DejaVuSans-65\" d=\"M 3597 1894 \n","L 3597 1613 \n","L 953 1613 \n","Q 991 1019 1311 708 \n","Q 1631 397 2203 397 \n","Q 2534 397 2845 478 \n","Q 3156 559 3463 722 \n","L 3463 178 \n","Q 3153 47 2828 -22 \n","Q 2503 -91 2169 -91 \n","Q 1331 -91 842 396 \n","Q 353 884 353 1716 \n","Q 353 2575 817 3079 \n","Q 1281 3584 2069 3584 \n","Q 2775 3584 3186 3129 \n","Q 3597 2675 3597 1894 \n","z\n","M 3022 2063 \n","Q 3016 2534 2758 2815 \n","Q 2500 3097 2075 3097 \n","Q 1594 3097 1305 2825 \n","Q 1016 2553 972 2059 \n","L 3022 2063 \n","z\n","\" transform=\"scale(0.015625)\"/>\n","       <path id=\"DejaVuSans-70\" d=\"M 1159 525 \n","L 1159 -1331 \n","L 581 -1331 \n","L 581 3500 \n","L 1159 3500 \n","L 1159 2969 \n","Q 1341 3281 1617 3432 \n","Q 1894 3584 2278 3584 \n","Q 2916 3584 3314 3078 \n","Q 3713 2572 3713 1747 \n","Q 3713 922 3314 415 \n","Q 2916 -91 2278 -91 \n","Q 1894 -91 1617 61 \n","Q 1341 213 1159 525 \n","z\n","M 3116 1747 \n","Q 3116 2381 2855 2742 \n","Q 2594 3103 2138 3103 \n","Q 1681 3103 1420 2742 \n","Q 1159 2381 1159 1747 \n","Q 1159 1113 1420 752 \n","Q 1681 391 2138 391 \n","Q 2594 391 2855 752 \n","Q 3116 1113 3116 1747 \n","z\n","\" transform=\"scale(0.015625)\"/>\n","       <path id=\"DejaVuSans-6f\" d=\"M 1959 3097 \n","Q 1497 3097 1228 2736 \n","Q 959 2375 959 1747 \n","Q 959 1119 1226 758 \n","Q 1494 397 1959 397 \n","Q 2419 397 2687 759 \n","Q 2956 1122 2956 1747 \n","Q 2956 2369 2687 2733 \n","Q 2419 3097 1959 3097 \n","z\n","M 1959 3584 \n","Q 2709 3584 3137 3096 \n","Q 3566 2609 3566 1747 \n","Q 3566 888 3137 398 \n","Q 2709 -91 1959 -91 \n","Q 1206 -91 779 398 \n","Q 353 888 353 1747 \n","Q 353 2609 779 3096 \n","Q 1206 3584 1959 3584 \n","z\n","\" transform=\"scale(0.015625)\"/>\n","       <path id=\"DejaVuSans-63\" d=\"M 3122 3366 \n","L 3122 2828 \n","Q 2878 2963 2633 3030 \n","Q 2388 3097 2138 3097 \n","Q 1578 3097 1268 2742 \n","Q 959 2388 959 1747 \n","Q 959 1106 1268 751 \n","Q 1578 397 2138 397 \n","Q 2388 397 2633 464 \n","Q 2878 531 3122 666 \n","L 3122 134 \n","Q 2881 22 2623 -34 \n","Q 2366 -91 2075 -91 \n","Q 1284 -91 818 406 \n","Q 353 903 353 1747 \n","Q 353 2603 823 3093 \n","Q 1294 3584 2113 3584 \n","Q 2378 3584 2631 3529 \n","Q 2884 3475 3122 3366 \n","z\n","\" transform=\"scale(0.015625)\"/>\n","       <path id=\"DejaVuSans-68\" d=\"M 3513 2113 \n","L 3513 0 \n","L 2938 0 \n","L 2938 2094 \n","Q 2938 2591 2744 2837 \n","Q 2550 3084 2163 3084 \n","Q 1697 3084 1428 2787 \n","Q 1159 2491 1159 1978 \n","L 1159 0 \n","L 581 0 \n","L 581 4863 \n","L 1159 4863 \n","L 1159 2956 \n","Q 1366 3272 1645 3428 \n","Q 1925 3584 2291 3584 \n","Q 2894 3584 3203 3211 \n","Q 3513 2838 3513 2113 \n","z\n","\" transform=\"scale(0.015625)\"/>\n","       <path id=\"DejaVuSans-73\" d=\"M 2834 3397 \n","L 2834 2853 \n","Q 2591 2978 2328 3040 \n","Q 2066 3103 1784 3103 \n","Q 1356 3103 1142 2972 \n","Q 928 2841 928 2578 \n","Q 928 2378 1081 2264 \n","Q 1234 2150 1697 2047 \n","L 1894 2003 \n","Q 2506 1872 2764 1633 \n","Q 3022 1394 3022 966 \n","Q 3022 478 2636 193 \n","Q 2250 -91 1575 -91 \n","Q 1294 -91 989 -36 \n","Q 684 19 347 128 \n","L 347 722 \n","Q 666 556 975 473 \n","Q 1284 391 1588 391 \n","Q 1994 391 2212 530 \n","Q 2431 669 2431 922 \n","Q 2431 1156 2273 1281 \n","Q 2116 1406 1581 1522 \n","L 1381 1569 \n","Q 847 1681 609 1914 \n","Q 372 2147 372 2553 \n","Q 372 3047 722 3315 \n","Q 1072 3584 1716 3584 \n","Q 2034 3584 2315 3537 \n","Q 2597 3491 2834 3397 \n","z\n","\" transform=\"scale(0.015625)\"/>\n","      </defs>\n","      <use xlink:href=\"#DejaVuSans-65\"/>\n","      <use xlink:href=\"#DejaVuSans-70\" x=\"61.523438\"/>\n","      <use xlink:href=\"#DejaVuSans-6f\" x=\"125\"/>\n","      <use xlink:href=\"#DejaVuSans-63\" x=\"186.181641\"/>\n","      <use xlink:href=\"#DejaVuSans-68\" x=\"241.162109\"/>\n","      <use xlink:href=\"#DejaVuSans-73\" x=\"304.541016\"/>\n","     </g>\n","    </g>\n","   </g>\n","   <g id=\"matplotlib.axis_2\">\n","    <g id=\"ytick_1\">\n","     <g id=\"line2d_11\">\n","      <path d=\"M 51.378125 119.277003 \n","L 246.678125 119.277003 \n","\" clip-path=\"url(#p5ed042e645)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n","     </g>\n","     <g id=\"line2d_12\">\n","      <defs>\n","       <path id=\"mb887f30291\" d=\"M 0 0 \n","L -3.5 0 \n","\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n","      </defs>\n","      <g>\n","       <use xlink:href=\"#mb887f30291\" x=\"51.378125\" y=\"119.277003\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n","      </g>\n","     </g>\n","     <g id=\"text_7\">\n","      <!-- $\\mathdefault{10^{-9}}$ -->\n","      <g transform=\"translate(20.878125 123.076222)scale(0.1 -0.1)\">\n","       <defs>\n","        <path id=\"DejaVuSans-2212\" d=\"M 678 2272 \n","L 4684 2272 \n","L 4684 1741 \n","L 678 1741 \n","L 678 2272 \n","z\n","\" transform=\"scale(0.015625)\"/>\n","        <path id=\"DejaVuSans-39\" d=\"M 703 97 \n","L 703 672 \n","Q 941 559 1184 500 \n","Q 1428 441 1663 441 \n","Q 2288 441 2617 861 \n","Q 2947 1281 2994 2138 \n","Q 2813 1869 2534 1725 \n","Q 2256 1581 1919 1581 \n","Q 1219 1581 811 2004 \n","Q 403 2428 403 3163 \n","Q 403 3881 828 4315 \n","Q 1253 4750 1959 4750 \n","Q 2769 4750 3195 4129 \n","Q 3622 3509 3622 2328 \n","Q 3622 1225 3098 567 \n","Q 2575 -91 1691 -91 \n","Q 1453 -91 1209 -44 \n","Q 966 3 703 97 \n","z\n","M 1959 2075 \n","Q 2384 2075 2632 2365 \n","Q 2881 2656 2881 3163 \n","Q 2881 3666 2632 3958 \n","Q 2384 4250 1959 4250 \n","Q 1534 4250 1286 3958 \n","Q 1038 3666 1038 3163 \n","Q 1038 2656 1286 2365 \n","Q 1534 2075 1959 2075 \n","z\n","\" transform=\"scale(0.015625)\"/>\n","       </defs>\n","       <use xlink:href=\"#DejaVuSans-31\" transform=\"translate(0 0.765625)\"/>\n","       <use xlink:href=\"#DejaVuSans-30\" transform=\"translate(63.623047 0.765625)\"/>\n","       <use xlink:href=\"#DejaVuSans-2212\" transform=\"translate(128.203125 39.046875)scale(0.7)\"/>\n","       <use xlink:href=\"#DejaVuSans-39\" transform=\"translate(186.855469 39.046875)scale(0.7)\"/>\n","      </g>\n","     </g>\n","    </g>\n","    <g id=\"ytick_2\">\n","     <g id=\"line2d_13\">\n","      <path d=\"M 51.378125 91.178407 \n","L 246.678125 91.178407 \n","\" clip-path=\"url(#p5ed042e645)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n","     </g>\n","     <g id=\"line2d_14\">\n","      <g>\n","       <use xlink:href=\"#mb887f30291\" x=\"51.378125\" y=\"91.178407\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n","      </g>\n","     </g>\n","     <g id=\"text_8\">\n","      <!-- $\\mathdefault{10^{-6}}$ -->\n","      <g transform=\"translate(20.878125 94.977626)scale(0.1 -0.1)\">\n","       <use xlink:href=\"#DejaVuSans-31\" transform=\"translate(0 0.765625)\"/>\n","       <use xlink:href=\"#DejaVuSans-30\" transform=\"translate(63.623047 0.765625)\"/>\n","       <use xlink:href=\"#DejaVuSans-2212\" transform=\"translate(128.203125 39.046875)scale(0.7)\"/>\n","       <use xlink:href=\"#DejaVuSans-36\" transform=\"translate(186.855469 39.046875)scale(0.7)\"/>\n","      </g>\n","     </g>\n","    </g>\n","    <g id=\"ytick_3\">\n","     <g id=\"line2d_15\">\n","      <path d=\"M 51.378125 63.079811 \n","L 246.678125 63.079811 \n","\" clip-path=\"url(#p5ed042e645)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n","     </g>\n","     <g id=\"line2d_16\">\n","      <g>\n","       <use xlink:href=\"#mb887f30291\" x=\"51.378125\" y=\"63.079811\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n","      </g>\n","     </g>\n","     <g id=\"text_9\">\n","      <!-- $\\mathdefault{10^{-3}}$ -->\n","      <g transform=\"translate(20.878125 66.87903)scale(0.1 -0.1)\">\n","       <defs>\n","        <path id=\"DejaVuSans-33\" d=\"M 2597 2516 \n","Q 3050 2419 3304 2112 \n","Q 3559 1806 3559 1356 \n","Q 3559 666 3084 287 \n","Q 2609 -91 1734 -91 \n","Q 1441 -91 1130 -33 \n","Q 819 25 488 141 \n","L 488 750 \n","Q 750 597 1062 519 \n","Q 1375 441 1716 441 \n","Q 2309 441 2620 675 \n","Q 2931 909 2931 1356 \n","Q 2931 1769 2642 2001 \n","Q 2353 2234 1838 2234 \n","L 1294 2234 \n","L 1294 2753 \n","L 1863 2753 \n","Q 2328 2753 2575 2939 \n","Q 2822 3125 2822 3475 \n","Q 2822 3834 2567 4026 \n","Q 2313 4219 1838 4219 \n","Q 1578 4219 1281 4162 \n","Q 984 4106 628 3988 \n","L 628 4550 \n","Q 988 4650 1302 4700 \n","Q 1616 4750 1894 4750 \n","Q 2613 4750 3031 4423 \n","Q 3450 4097 3450 3541 \n","Q 3450 3153 3228 2886 \n","Q 3006 2619 2597 2516 \n","z\n","\" transform=\"scale(0.015625)\"/>\n","       </defs>\n","       <use xlink:href=\"#DejaVuSans-31\" transform=\"translate(0 0.765625)\"/>\n","       <use xlink:href=\"#DejaVuSans-30\" transform=\"translate(63.623047 0.765625)\"/>\n","       <use xlink:href=\"#DejaVuSans-2212\" transform=\"translate(128.203125 39.046875)scale(0.7)\"/>\n","       <use xlink:href=\"#DejaVuSans-33\" transform=\"translate(186.855469 39.046875)scale(0.7)\"/>\n","      </g>\n","     </g>\n","    </g>\n","    <g id=\"ytick_4\">\n","     <g id=\"line2d_17\">\n","      <path d=\"M 51.378125 34.981215 \n","L 246.678125 34.981215 \n","\" clip-path=\"url(#p5ed042e645)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n","     </g>\n","     <g id=\"line2d_18\">\n","      <g>\n","       <use xlink:href=\"#mb887f30291\" x=\"51.378125\" y=\"34.981215\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n","      </g>\n","     </g>\n","     <g id=\"text_10\">\n","      <!-- $\\mathdefault{10^{0}}$ -->\n","      <g transform=\"translate(26.778125 38.780434)scale(0.1 -0.1)\">\n","       <use xlink:href=\"#DejaVuSans-31\" transform=\"translate(0 0.765625)\"/>\n","       <use xlink:href=\"#DejaVuSans-30\" transform=\"translate(63.623047 0.765625)\"/>\n","       <use xlink:href=\"#DejaVuSans-30\" transform=\"translate(128.203125 39.046875)scale(0.7)\"/>\n","      </g>\n","     </g>\n","    </g>\n","    <g id=\"text_11\">\n","     <!-- loss -->\n","     <g transform=\"translate(14.798437 84.807812)rotate(-90)scale(0.1 -0.1)\">\n","      <defs>\n","       <path id=\"DejaVuSans-6c\" d=\"M 603 4863 \n","L 1178 4863 \n","L 1178 0 \n","L 603 0 \n","L 603 4863 \n","z\n","\" transform=\"scale(0.015625)\"/>\n","      </defs>\n","      <use xlink:href=\"#DejaVuSans-6c\"/>\n","      <use xlink:href=\"#DejaVuSans-6f\" x=\"27.783203\"/>\n","      <use xlink:href=\"#DejaVuSans-73\" x=\"88.964844\"/>\n","      <use xlink:href=\"#DejaVuSans-73\" x=\"141.064453\"/>\n","     </g>\n","    </g>\n","   </g>\n","   <g id=\"line2d_19\">\n","    <path d=\"M 51.378125 24.168093 \n","L 61.657072 33.711551 \n","L 71.93602 41.915027 \n","L 82.214967 49.524061 \n","L 92.493914 56.738217 \n","L 102.772862 63.769639 \n","L 113.051809 70.64042 \n","L 123.330757 77.402652 \n","L 133.609704 84.085355 \n","L 143.888651 90.66234 \n","L 154.167599 97.107542 \n","L 164.446546 103.43541 \n","L 174.725493 109.653939 \n","L 185.004441 115.661614 \n","L 195.283388 121.408282 \n","L 205.562336 126.776031 \n","L 215.841283 130.811171 \n","L 226.12023 133.648442 \n","L 236.399178 135.602606 \n","L 246.678125 136.922727 \n","\" clip-path=\"url(#p5ed042e645)\" style=\"fill: none; stroke: #1f77b4; stroke-width: 1.5; stroke-linecap: square\"/>\n","   </g>\n","   <g id=\"line2d_20\">\n","    <path d=\"M 51.378125 13.377273 \n","L 61.657072 13.425898 \n","L 71.93602 13.447246 \n","L 82.214967 13.456447 \n","L 92.493914 13.460139 \n","L 102.772862 13.461661 \n","L 113.051809 13.462259 \n","L 123.330757 13.462501 \n","L 133.609704 13.462598 \n","L 143.888651 13.462637 \n","L 154.167599 13.462652 \n","L 164.446546 13.462658 \n","L 174.725493 13.462661 \n","L 185.004441 13.462662 \n","L 195.283388 13.462662 \n","L 205.562336 13.462662 \n","L 215.841283 13.462662 \n","L 226.12023 13.462662 \n","L 236.399178 13.462662 \n","L 246.678125 13.462662 \n","\" clip-path=\"url(#p5ed042e645)\" style=\"fill: none; stroke-dasharray: 5.55,2.4; stroke-dashoffset: 0; stroke: #bf00bf; stroke-width: 1.5\"/>\n","   </g>\n","   <g id=\"patch_3\">\n","    <path d=\"M 51.378125 143.1 \n","L 51.378125 7.2 \n","\" style=\"fill: none; stroke: #000000; stroke-width: 0.8; stroke-linejoin: miter; stroke-linecap: square\"/>\n","   </g>\n","   <g id=\"patch_4\">\n","    <path d=\"M 246.678125 143.1 \n","L 246.678125 7.2 \n","\" style=\"fill: none; stroke: #000000; stroke-width: 0.8; stroke-linejoin: miter; stroke-linecap: square\"/>\n","   </g>\n","   <g id=\"patch_5\">\n","    <path d=\"M 51.378125 143.1 \n","L 246.678125 143.1 \n","\" style=\"fill: none; stroke: #000000; stroke-width: 0.8; stroke-linejoin: miter; stroke-linecap: square\"/>\n","   </g>\n","   <g id=\"patch_6\">\n","    <path d=\"M 51.378125 7.2 \n","L 246.678125 7.2 \n","\" style=\"fill: none; stroke: #000000; stroke-width: 0.8; stroke-linejoin: miter; stroke-linecap: square\"/>\n","   </g>\n","   <g id=\"legend_1\">\n","    <g id=\"patch_7\">\n","     <path d=\"M 58.378125 138.1 \n","L 113.653125 138.1 \n","Q 115.653125 138.1 115.653125 136.1 \n","L 115.653125 107.74375 \n","Q 115.653125 105.74375 113.653125 105.74375 \n","L 58.378125 105.74375 \n","Q 56.378125 105.74375 56.378125 107.74375 \n","L 56.378125 136.1 \n","Q 56.378125 138.1 58.378125 138.1 \n","z\n","\" style=\"fill: #ffffff; opacity: 0.8; stroke: #cccccc; stroke-linejoin: miter\"/>\n","    </g>\n","    <g id=\"line2d_21\">\n","     <path d=\"M 60.378125 113.842187 \n","L 70.378125 113.842187 \n","L 80.378125 113.842187 \n","\" style=\"fill: none; stroke: #1f77b4; stroke-width: 1.5; stroke-linecap: square\"/>\n","    </g>\n","    <g id=\"text_12\">\n","     <!-- train -->\n","     <g transform=\"translate(88.378125 117.342187)scale(0.1 -0.1)\">\n","      <defs>\n","       <path id=\"DejaVuSans-74\" d=\"M 1172 4494 \n","L 1172 3500 \n","L 2356 3500 \n","L 2356 3053 \n","L 1172 3053 \n","L 1172 1153 \n","Q 1172 725 1289 603 \n","Q 1406 481 1766 481 \n","L 2356 481 \n","L 2356 0 \n","L 1766 0 \n","Q 1100 0 847 248 \n","Q 594 497 594 1153 \n","L 594 3053 \n","L 172 3053 \n","L 172 3500 \n","L 594 3500 \n","L 594 4494 \n","L 1172 4494 \n","z\n","\" transform=\"scale(0.015625)\"/>\n","       <path id=\"DejaVuSans-72\" d=\"M 2631 2963 \n","Q 2534 3019 2420 3045 \n","Q 2306 3072 2169 3072 \n","Q 1681 3072 1420 2755 \n","Q 1159 2438 1159 1844 \n","L 1159 0 \n","L 581 0 \n","L 581 3500 \n","L 1159 3500 \n","L 1159 2956 \n","Q 1341 3275 1631 3429 \n","Q 1922 3584 2338 3584 \n","Q 2397 3584 2469 3576 \n","Q 2541 3569 2628 3553 \n","L 2631 2963 \n","z\n","\" transform=\"scale(0.015625)\"/>\n","       <path id=\"DejaVuSans-61\" d=\"M 2194 1759 \n","Q 1497 1759 1228 1600 \n","Q 959 1441 959 1056 \n","Q 959 750 1161 570 \n","Q 1363 391 1709 391 \n","Q 2188 391 2477 730 \n","Q 2766 1069 2766 1631 \n","L 2766 1759 \n","L 2194 1759 \n","z\n","M 3341 1997 \n","L 3341 0 \n","L 2766 0 \n","L 2766 531 \n","Q 2569 213 2275 61 \n","Q 1981 -91 1556 -91 \n","Q 1019 -91 701 211 \n","Q 384 513 384 1019 \n","Q 384 1609 779 1909 \n","Q 1175 2209 1959 2209 \n","L 2766 2209 \n","L 2766 2266 \n","Q 2766 2663 2505 2880 \n","Q 2244 3097 1772 3097 \n","Q 1472 3097 1187 3025 \n","Q 903 2953 641 2809 \n","L 641 3341 \n","Q 956 3463 1253 3523 \n","Q 1550 3584 1831 3584 \n","Q 2591 3584 2966 3190 \n","Q 3341 2797 3341 1997 \n","z\n","\" transform=\"scale(0.015625)\"/>\n","       <path id=\"DejaVuSans-69\" d=\"M 603 3500 \n","L 1178 3500 \n","L 1178 0 \n","L 603 0 \n","L 603 3500 \n","z\n","M 603 4863 \n","L 1178 4863 \n","L 1178 4134 \n","L 603 4134 \n","L 603 4863 \n","z\n","\" transform=\"scale(0.015625)\"/>\n","       <path id=\"DejaVuSans-6e\" d=\"M 3513 2113 \n","L 3513 0 \n","L 2938 0 \n","L 2938 2094 \n","Q 2938 2591 2744 2837 \n","Q 2550 3084 2163 3084 \n","Q 1697 3084 1428 2787 \n","Q 1159 2491 1159 1978 \n","L 1159 0 \n","L 581 0 \n","L 581 3500 \n","L 1159 3500 \n","L 1159 2956 \n","Q 1366 3272 1645 3428 \n","Q 1925 3584 2291 3584 \n","Q 2894 3584 3203 3211 \n","Q 3513 2838 3513 2113 \n","z\n","\" transform=\"scale(0.015625)\"/>\n","      </defs>\n","      <use xlink:href=\"#DejaVuSans-74\"/>\n","      <use xlink:href=\"#DejaVuSans-72\" x=\"39.208984\"/>\n","      <use xlink:href=\"#DejaVuSans-61\" x=\"80.322266\"/>\n","      <use xlink:href=\"#DejaVuSans-69\" x=\"141.601562\"/>\n","      <use xlink:href=\"#DejaVuSans-6e\" x=\"169.384766\"/>\n","     </g>\n","    </g>\n","    <g id=\"line2d_22\">\n","     <path d=\"M 60.378125 128.520312 \n","L 70.378125 128.520312 \n","L 80.378125 128.520312 \n","\" style=\"fill: none; stroke-dasharray: 5.55,2.4; stroke-dashoffset: 0; stroke: #bf00bf; stroke-width: 1.5\"/>\n","    </g>\n","    <g id=\"text_13\">\n","     <!-- test -->\n","     <g transform=\"translate(88.378125 132.020312)scale(0.1 -0.1)\">\n","      <use xlink:href=\"#DejaVuSans-74\"/>\n","      <use xlink:href=\"#DejaVuSans-65\" x=\"39.208984\"/>\n","      <use xlink:href=\"#DejaVuSans-73\" x=\"100.732422\"/>\n","      <use xlink:href=\"#DejaVuSans-74\" x=\"152.832031\"/>\n","     </g>\n","    </g>\n","   </g>\n","  </g>\n"," </g>\n"," <defs>\n","  <clipPath id=\"p5ed042e645\">\n","   <rect x=\"51.378125\" y=\"7.2\" width=\"195.3\" height=\"135.9\"/>\n","  </clipPath>\n"," </defs>\n","</svg>\n"],"text/plain":["<Figure size 252x180 with 1 Axes>"]},"metadata":{"needs_background":"light"},"output_type":"display_data"}],"source":["train_concise(0)"],"id":"7vqPr0M3hFj2"},{"cell_type":"code","execution_count":null,"metadata":{"execution":{"iopub.execute_input":"2022-07-31T02:23:07.968650Z","iopub.status.busy":"2022-07-31T02:23:07.968162Z","iopub.status.idle":"2022-07-31T02:23:17.124533Z","shell.execute_reply":"2022-07-31T02:23:17.123862Z"},"origin_pos":31,"tab":["pytorch"],"outputId":"344001d5-8b4a-4ad7-98eb-ab0dd4d122d2","id":"lQO9u3UVhFj3"},"outputs":[{"name":"stdout","output_type":"stream","text":["w的L2范数： 0.7186794877052307\n"]},{"data":{"image/svg+xml":["<?xml version=\"1.0\" encoding=\"utf-8\" standalone=\"no\"?>\n","<!DOCTYPE svg PUBLIC \"-//W3C//DTD SVG 1.1//EN\"\n","  \"http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd\">\n","<svg xmlns:xlink=\"http://www.w3.org/1999/xlink\" width=\"263.421875pt\" height=\"180.65625pt\" viewBox=\"0 0 263.421875 180.65625\" xmlns=\"http://www.w3.org/2000/svg\" version=\"1.1\">\n"," <metadata>\n","  <rdf:RDF xmlns:dc=\"http://purl.org/dc/elements/1.1/\" xmlns:cc=\"http://creativecommons.org/ns#\" xmlns:rdf=\"http://www.w3.org/1999/02/22-rdf-syntax-ns#\">\n","   <cc:Work>\n","    <dc:type rdf:resource=\"http://purl.org/dc/dcmitype/StillImage\"/>\n","    <dc:date>2022-07-31T02:23:17.007647</dc:date>\n","    <dc:format>image/svg+xml</dc:format>\n","    <dc:creator>\n","     <cc:Agent>\n","      <dc:title>Matplotlib v3.5.1, https://matplotlib.org/</dc:title>\n","     </cc:Agent>\n","    </dc:creator>\n","   </cc:Work>\n","  </rdf:RDF>\n"," </metadata>\n"," <defs>\n","  <style type=\"text/css\">*{stroke-linejoin: round; stroke-linecap: butt}</style>\n"," </defs>\n"," <g id=\"figure_1\">\n","  <g id=\"patch_1\">\n","   <path d=\"M 0 180.65625 \n","L 263.421875 180.65625 \n","L 263.421875 0 \n","L 0 0 \n","L 0 180.65625 \n","z\n","\" style=\"fill: none\"/>\n","  </g>\n","  <g id=\"axes_1\">\n","   <g id=\"patch_2\">\n","    <path d=\"M 51.378125 143.1 \n","L 246.678125 143.1 \n","L 246.678125 7.2 \n","L 51.378125 7.2 \n","z\n","\" style=\"fill: #ffffff\"/>\n","   </g>\n","   <g id=\"matplotlib.axis_1\">\n","    <g id=\"xtick_1\">\n","     <g id=\"line2d_1\">\n","      <path d=\"M 82.214967 143.1 \n","L 82.214967 7.2 \n","\" clip-path=\"url(#p97ac790d64)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n","     </g>\n","     <g id=\"line2d_2\">\n","      <defs>\n","       <path id=\"m4856621ce4\" d=\"M 0 0 \n","L 0 3.5 \n","\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n","      </defs>\n","      <g>\n","       <use xlink:href=\"#m4856621ce4\" x=\"82.214967\" y=\"143.1\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n","      </g>\n","     </g>\n","     <g id=\"text_1\">\n","      <!-- 20 -->\n","      <g transform=\"translate(75.852467 157.698438)scale(0.1 -0.1)\">\n","       <defs>\n","        <path id=\"DejaVuSans-32\" d=\"M 1228 531 \n","L 3431 531 \n","L 3431 0 \n","L 469 0 \n","L 469 531 \n","Q 828 903 1448 1529 \n","Q 2069 2156 2228 2338 \n","Q 2531 2678 2651 2914 \n","Q 2772 3150 2772 3378 \n","Q 2772 3750 2511 3984 \n","Q 2250 4219 1831 4219 \n","Q 1534 4219 1204 4116 \n","Q 875 4013 500 3803 \n","L 500 4441 \n","Q 881 4594 1212 4672 \n","Q 1544 4750 1819 4750 \n","Q 2544 4750 2975 4387 \n","Q 3406 4025 3406 3419 \n","Q 3406 3131 3298 2873 \n","Q 3191 2616 2906 2266 \n","Q 2828 2175 2409 1742 \n","Q 1991 1309 1228 531 \n","z\n","\" transform=\"scale(0.015625)\"/>\n","        <path id=\"DejaVuSans-30\" d=\"M 2034 4250 \n","Q 1547 4250 1301 3770 \n","Q 1056 3291 1056 2328 \n","Q 1056 1369 1301 889 \n","Q 1547 409 2034 409 \n","Q 2525 409 2770 889 \n","Q 3016 1369 3016 2328 \n","Q 3016 3291 2770 3770 \n","Q 2525 4250 2034 4250 \n","z\n","M 2034 4750 \n","Q 2819 4750 3233 4129 \n","Q 3647 3509 3647 2328 \n","Q 3647 1150 3233 529 \n","Q 2819 -91 2034 -91 \n","Q 1250 -91 836 529 \n","Q 422 1150 422 2328 \n","Q 422 3509 836 4129 \n","Q 1250 4750 2034 4750 \n","z\n","\" transform=\"scale(0.015625)\"/>\n","       </defs>\n","       <use xlink:href=\"#DejaVuSans-32\"/>\n","       <use xlink:href=\"#DejaVuSans-30\" x=\"63.623047\"/>\n","      </g>\n","     </g>\n","    </g>\n","    <g id=\"xtick_2\">\n","     <g id=\"line2d_3\">\n","      <path d=\"M 123.330757 143.1 \n","L 123.330757 7.2 \n","\" clip-path=\"url(#p97ac790d64)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n","     </g>\n","     <g id=\"line2d_4\">\n","      <g>\n","       <use xlink:href=\"#m4856621ce4\" x=\"123.330757\" y=\"143.1\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n","      </g>\n","     </g>\n","     <g id=\"text_2\">\n","      <!-- 40 -->\n","      <g transform=\"translate(116.968257 157.698438)scale(0.1 -0.1)\">\n","       <defs>\n","        <path id=\"DejaVuSans-34\" d=\"M 2419 4116 \n","L 825 1625 \n","L 2419 1625 \n","L 2419 4116 \n","z\n","M 2253 4666 \n","L 3047 4666 \n","L 3047 1625 \n","L 3713 1625 \n","L 3713 1100 \n","L 3047 1100 \n","L 3047 0 \n","L 2419 0 \n","L 2419 1100 \n","L 313 1100 \n","L 313 1709 \n","L 2253 4666 \n","z\n","\" transform=\"scale(0.015625)\"/>\n","       </defs>\n","       <use xlink:href=\"#DejaVuSans-34\"/>\n","       <use xlink:href=\"#DejaVuSans-30\" x=\"63.623047\"/>\n","      </g>\n","     </g>\n","    </g>\n","    <g id=\"xtick_3\">\n","     <g id=\"line2d_5\">\n","      <path d=\"M 164.446546 143.1 \n","L 164.446546 7.2 \n","\" clip-path=\"url(#p97ac790d64)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n","     </g>\n","     <g id=\"line2d_6\">\n","      <g>\n","       <use xlink:href=\"#m4856621ce4\" x=\"164.446546\" y=\"143.1\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n","      </g>\n","     </g>\n","     <g id=\"text_3\">\n","      <!-- 60 -->\n","      <g transform=\"translate(158.084046 157.698438)scale(0.1 -0.1)\">\n","       <defs>\n","        <path id=\"DejaVuSans-36\" d=\"M 2113 2584 \n","Q 1688 2584 1439 2293 \n","Q 1191 2003 1191 1497 \n","Q 1191 994 1439 701 \n","Q 1688 409 2113 409 \n","Q 2538 409 2786 701 \n","Q 3034 994 3034 1497 \n","Q 3034 2003 2786 2293 \n","Q 2538 2584 2113 2584 \n","z\n","M 3366 4563 \n","L 3366 3988 \n","Q 3128 4100 2886 4159 \n","Q 2644 4219 2406 4219 \n","Q 1781 4219 1451 3797 \n","Q 1122 3375 1075 2522 \n","Q 1259 2794 1537 2939 \n","Q 1816 3084 2150 3084 \n","Q 2853 3084 3261 2657 \n","Q 3669 2231 3669 1497 \n","Q 3669 778 3244 343 \n","Q 2819 -91 2113 -91 \n","Q 1303 -91 875 529 \n","Q 447 1150 447 2328 \n","Q 447 3434 972 4092 \n","Q 1497 4750 2381 4750 \n","Q 2619 4750 2861 4703 \n","Q 3103 4656 3366 4563 \n","z\n","\" transform=\"scale(0.015625)\"/>\n","       </defs>\n","       <use xlink:href=\"#DejaVuSans-36\"/>\n","       <use xlink:href=\"#DejaVuSans-30\" x=\"63.623047\"/>\n","      </g>\n","     </g>\n","    </g>\n","    <g id=\"xtick_4\">\n","     <g id=\"line2d_7\">\n","      <path d=\"M 205.562336 143.1 \n","L 205.562336 7.2 \n","\" clip-path=\"url(#p97ac790d64)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n","     </g>\n","     <g id=\"line2d_8\">\n","      <g>\n","       <use xlink:href=\"#m4856621ce4\" x=\"205.562336\" y=\"143.1\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n","      </g>\n","     </g>\n","     <g id=\"text_4\">\n","      <!-- 80 -->\n","      <g transform=\"translate(199.199836 157.698438)scale(0.1 -0.1)\">\n","       <defs>\n","        <path id=\"DejaVuSans-38\" d=\"M 2034 2216 \n","Q 1584 2216 1326 1975 \n","Q 1069 1734 1069 1313 \n","Q 1069 891 1326 650 \n","Q 1584 409 2034 409 \n","Q 2484 409 2743 651 \n","Q 3003 894 3003 1313 \n","Q 3003 1734 2745 1975 \n","Q 2488 2216 2034 2216 \n","z\n","M 1403 2484 \n","Q 997 2584 770 2862 \n","Q 544 3141 544 3541 \n","Q 544 4100 942 4425 \n","Q 1341 4750 2034 4750 \n","Q 2731 4750 3128 4425 \n","Q 3525 4100 3525 3541 \n","Q 3525 3141 3298 2862 \n","Q 3072 2584 2669 2484 \n","Q 3125 2378 3379 2068 \n","Q 3634 1759 3634 1313 \n","Q 3634 634 3220 271 \n","Q 2806 -91 2034 -91 \n","Q 1263 -91 848 271 \n","Q 434 634 434 1313 \n","Q 434 1759 690 2068 \n","Q 947 2378 1403 2484 \n","z\n","M 1172 3481 \n","Q 1172 3119 1398 2916 \n","Q 1625 2713 2034 2713 \n","Q 2441 2713 2670 2916 \n","Q 2900 3119 2900 3481 \n","Q 2900 3844 2670 4047 \n","Q 2441 4250 2034 4250 \n","Q 1625 4250 1398 4047 \n","Q 1172 3844 1172 3481 \n","z\n","\" transform=\"scale(0.015625)\"/>\n","       </defs>\n","       <use xlink:href=\"#DejaVuSans-38\"/>\n","       <use xlink:href=\"#DejaVuSans-30\" x=\"63.623047\"/>\n","      </g>\n","     </g>\n","    </g>\n","    <g id=\"xtick_5\">\n","     <g id=\"line2d_9\">\n","      <path d=\"M 246.678125 143.1 \n","L 246.678125 7.2 \n","\" clip-path=\"url(#p97ac790d64)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n","     </g>\n","     <g id=\"line2d_10\">\n","      <g>\n","       <use xlink:href=\"#m4856621ce4\" x=\"246.678125\" y=\"143.1\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n","      </g>\n","     </g>\n","     <g id=\"text_5\">\n","      <!-- 100 -->\n","      <g transform=\"translate(237.134375 157.698438)scale(0.1 -0.1)\">\n","       <defs>\n","        <path id=\"DejaVuSans-31\" d=\"M 794 531 \n","L 1825 531 \n","L 1825 4091 \n","L 703 3866 \n","L 703 4441 \n","L 1819 4666 \n","L 2450 4666 \n","L 2450 531 \n","L 3481 531 \n","L 3481 0 \n","L 794 0 \n","L 794 531 \n","z\n","\" transform=\"scale(0.015625)\"/>\n","       </defs>\n","       <use xlink:href=\"#DejaVuSans-31\"/>\n","       <use xlink:href=\"#DejaVuSans-30\" x=\"63.623047\"/>\n","       <use xlink:href=\"#DejaVuSans-30\" x=\"127.246094\"/>\n","      </g>\n","     </g>\n","    </g>\n","    <g id=\"text_6\">\n","     <!-- epochs -->\n","     <g transform=\"translate(131.195312 171.376563)scale(0.1 -0.1)\">\n","      <defs>\n","       <path id=\"DejaVuSans-65\" d=\"M 3597 1894 \n","L 3597 1613 \n","L 953 1613 \n","Q 991 1019 1311 708 \n","Q 1631 397 2203 397 \n","Q 2534 397 2845 478 \n","Q 3156 559 3463 722 \n","L 3463 178 \n","Q 3153 47 2828 -22 \n","Q 2503 -91 2169 -91 \n","Q 1331 -91 842 396 \n","Q 353 884 353 1716 \n","Q 353 2575 817 3079 \n","Q 1281 3584 2069 3584 \n","Q 2775 3584 3186 3129 \n","Q 3597 2675 3597 1894 \n","z\n","M 3022 2063 \n","Q 3016 2534 2758 2815 \n","Q 2500 3097 2075 3097 \n","Q 1594 3097 1305 2825 \n","Q 1016 2553 972 2059 \n","L 3022 2063 \n","z\n","\" transform=\"scale(0.015625)\"/>\n","       <path id=\"DejaVuSans-70\" d=\"M 1159 525 \n","L 1159 -1331 \n","L 581 -1331 \n","L 581 3500 \n","L 1159 3500 \n","L 1159 2969 \n","Q 1341 3281 1617 3432 \n","Q 1894 3584 2278 3584 \n","Q 2916 3584 3314 3078 \n","Q 3713 2572 3713 1747 \n","Q 3713 922 3314 415 \n","Q 2916 -91 2278 -91 \n","Q 1894 -91 1617 61 \n","Q 1341 213 1159 525 \n","z\n","M 3116 1747 \n","Q 3116 2381 2855 2742 \n","Q 2594 3103 2138 3103 \n","Q 1681 3103 1420 2742 \n","Q 1159 2381 1159 1747 \n","Q 1159 1113 1420 752 \n","Q 1681 391 2138 391 \n","Q 2594 391 2855 752 \n","Q 3116 1113 3116 1747 \n","z\n","\" transform=\"scale(0.015625)\"/>\n","       <path id=\"DejaVuSans-6f\" d=\"M 1959 3097 \n","Q 1497 3097 1228 2736 \n","Q 959 2375 959 1747 \n","Q 959 1119 1226 758 \n","Q 1494 397 1959 397 \n","Q 2419 397 2687 759 \n","Q 2956 1122 2956 1747 \n","Q 2956 2369 2687 2733 \n","Q 2419 3097 1959 3097 \n","z\n","M 1959 3584 \n","Q 2709 3584 3137 3096 \n","Q 3566 2609 3566 1747 \n","Q 3566 888 3137 398 \n","Q 2709 -91 1959 -91 \n","Q 1206 -91 779 398 \n","Q 353 888 353 1747 \n","Q 353 2609 779 3096 \n","Q 1206 3584 1959 3584 \n","z\n","\" transform=\"scale(0.015625)\"/>\n","       <path id=\"DejaVuSans-63\" d=\"M 3122 3366 \n","L 3122 2828 \n","Q 2878 2963 2633 3030 \n","Q 2388 3097 2138 3097 \n","Q 1578 3097 1268 2742 \n","Q 959 2388 959 1747 \n","Q 959 1106 1268 751 \n","Q 1578 397 2138 397 \n","Q 2388 397 2633 464 \n","Q 2878 531 3122 666 \n","L 3122 134 \n","Q 2881 22 2623 -34 \n","Q 2366 -91 2075 -91 \n","Q 1284 -91 818 406 \n","Q 353 903 353 1747 \n","Q 353 2603 823 3093 \n","Q 1294 3584 2113 3584 \n","Q 2378 3584 2631 3529 \n","Q 2884 3475 3122 3366 \n","z\n","\" transform=\"scale(0.015625)\"/>\n","       <path id=\"DejaVuSans-68\" d=\"M 3513 2113 \n","L 3513 0 \n","L 2938 0 \n","L 2938 2094 \n","Q 2938 2591 2744 2837 \n","Q 2550 3084 2163 3084 \n","Q 1697 3084 1428 2787 \n","Q 1159 2491 1159 1978 \n","L 1159 0 \n","L 581 0 \n","L 581 4863 \n","L 1159 4863 \n","L 1159 2956 \n","Q 1366 3272 1645 3428 \n","Q 1925 3584 2291 3584 \n","Q 2894 3584 3203 3211 \n","Q 3513 2838 3513 2113 \n","z\n","\" transform=\"scale(0.015625)\"/>\n","       <path id=\"DejaVuSans-73\" d=\"M 2834 3397 \n","L 2834 2853 \n","Q 2591 2978 2328 3040 \n","Q 2066 3103 1784 3103 \n","Q 1356 3103 1142 2972 \n","Q 928 2841 928 2578 \n","Q 928 2378 1081 2264 \n","Q 1234 2150 1697 2047 \n","L 1894 2003 \n","Q 2506 1872 2764 1633 \n","Q 3022 1394 3022 966 \n","Q 3022 478 2636 193 \n","Q 2250 -91 1575 -91 \n","Q 1294 -91 989 -36 \n","Q 684 19 347 128 \n","L 347 722 \n","Q 666 556 975 473 \n","Q 1284 391 1588 391 \n","Q 1994 391 2212 530 \n","Q 2431 669 2431 922 \n","Q 2431 1156 2273 1281 \n","Q 2116 1406 1581 1522 \n","L 1381 1569 \n","Q 847 1681 609 1914 \n","Q 372 2147 372 2553 \n","Q 372 3047 722 3315 \n","Q 1072 3584 1716 3584 \n","Q 2034 3584 2315 3537 \n","Q 2597 3491 2834 3397 \n","z\n","\" transform=\"scale(0.015625)\"/>\n","      </defs>\n","      <use xlink:href=\"#DejaVuSans-65\"/>\n","      <use xlink:href=\"#DejaVuSans-70\" x=\"61.523438\"/>\n","      <use xlink:href=\"#DejaVuSans-6f\" x=\"125\"/>\n","      <use xlink:href=\"#DejaVuSans-63\" x=\"186.181641\"/>\n","      <use xlink:href=\"#DejaVuSans-68\" x=\"241.162109\"/>\n","      <use xlink:href=\"#DejaVuSans-73\" x=\"304.541016\"/>\n","     </g>\n","    </g>\n","   </g>\n","   <g id=\"matplotlib.axis_2\">\n","    <g id=\"ytick_1\">\n","     <g id=\"line2d_11\">\n","      <path d=\"M 51.378125 131.054124 \n","L 246.678125 131.054124 \n","\" clip-path=\"url(#p97ac790d64)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n","     </g>\n","     <g id=\"line2d_12\">\n","      <defs>\n","       <path id=\"m5d32c10c90\" d=\"M 0 0 \n","L -3.5 0 \n","\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n","      </defs>\n","      <g>\n","       <use xlink:href=\"#m5d32c10c90\" x=\"51.378125\" y=\"131.054124\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n","      </g>\n","     </g>\n","     <g id=\"text_7\">\n","      <!-- $\\mathdefault{10^{-1}}$ -->\n","      <g transform=\"translate(20.878125 134.853343)scale(0.1 -0.1)\">\n","       <defs>\n","        <path id=\"DejaVuSans-2212\" d=\"M 678 2272 \n","L 4684 2272 \n","L 4684 1741 \n","L 678 1741 \n","L 678 2272 \n","z\n","\" transform=\"scale(0.015625)\"/>\n","       </defs>\n","       <use xlink:href=\"#DejaVuSans-31\" transform=\"translate(0 0.684375)\"/>\n","       <use xlink:href=\"#DejaVuSans-30\" transform=\"translate(63.623047 0.684375)\"/>\n","       <use xlink:href=\"#DejaVuSans-2212\" transform=\"translate(128.203125 38.965625)scale(0.7)\"/>\n","       <use xlink:href=\"#DejaVuSans-31\" transform=\"translate(186.855469 38.965625)scale(0.7)\"/>\n","      </g>\n","     </g>\n","    </g>\n","    <g id=\"ytick_2\">\n","     <g id=\"line2d_13\">\n","      <path d=\"M 51.378125 95.094219 \n","L 246.678125 95.094219 \n","\" clip-path=\"url(#p97ac790d64)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n","     </g>\n","     <g id=\"line2d_14\">\n","      <g>\n","       <use xlink:href=\"#m5d32c10c90\" x=\"51.378125\" y=\"95.094219\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n","      </g>\n","     </g>\n","     <g id=\"text_8\">\n","      <!-- $\\mathdefault{10^{0}}$ -->\n","      <g transform=\"translate(26.778125 98.893437)scale(0.1 -0.1)\">\n","       <use xlink:href=\"#DejaVuSans-31\" transform=\"translate(0 0.765625)\"/>\n","       <use xlink:href=\"#DejaVuSans-30\" transform=\"translate(63.623047 0.765625)\"/>\n","       <use xlink:href=\"#DejaVuSans-30\" transform=\"translate(128.203125 39.046875)scale(0.7)\"/>\n","      </g>\n","     </g>\n","    </g>\n","    <g id=\"ytick_3\">\n","     <g id=\"line2d_15\">\n","      <path d=\"M 51.378125 59.134313 \n","L 246.678125 59.134313 \n","\" clip-path=\"url(#p97ac790d64)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n","     </g>\n","     <g id=\"line2d_16\">\n","      <g>\n","       <use xlink:href=\"#m5d32c10c90\" x=\"51.378125\" y=\"59.134313\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n","      </g>\n","     </g>\n","     <g id=\"text_9\">\n","      <!-- $\\mathdefault{10^{1}}$ -->\n","      <g transform=\"translate(26.778125 62.933532)scale(0.1 -0.1)\">\n","       <use xlink:href=\"#DejaVuSans-31\" transform=\"translate(0 0.684375)\"/>\n","       <use xlink:href=\"#DejaVuSans-30\" transform=\"translate(63.623047 0.684375)\"/>\n","       <use xlink:href=\"#DejaVuSans-31\" transform=\"translate(128.203125 38.965625)scale(0.7)\"/>\n","      </g>\n","     </g>\n","    </g>\n","    <g id=\"ytick_4\">\n","     <g id=\"line2d_17\">\n","      <path d=\"M 51.378125 23.174408 \n","L 246.678125 23.174408 \n","\" clip-path=\"url(#p97ac790d64)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n","     </g>\n","     <g id=\"line2d_18\">\n","      <g>\n","       <use xlink:href=\"#m5d32c10c90\" x=\"51.378125\" y=\"23.174408\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n","      </g>\n","     </g>\n","     <g id=\"text_10\">\n","      <!-- $\\mathdefault{10^{2}}$ -->\n","      <g transform=\"translate(26.778125 26.973626)scale(0.1 -0.1)\">\n","       <use xlink:href=\"#DejaVuSans-31\" transform=\"translate(0 0.765625)\"/>\n","       <use xlink:href=\"#DejaVuSans-30\" transform=\"translate(63.623047 0.765625)\"/>\n","       <use xlink:href=\"#DejaVuSans-32\" transform=\"translate(128.203125 39.046875)scale(0.7)\"/>\n","      </g>\n","     </g>\n","    </g>\n","    <g id=\"ytick_5\">\n","     <g id=\"line2d_19\">\n","      <defs>\n","       <path id=\"mfa9bd9e3c7\" d=\"M 0 0 \n","L -2 0 \n","\" style=\"stroke: #000000; stroke-width: 0.6\"/>\n","      </defs>\n","      <g>\n","       <use xlink:href=\"#mfa9bd9e3c7\" x=\"51.378125\" y=\"141.879135\" style=\"stroke: #000000; stroke-width: 0.6\"/>\n","      </g>\n","     </g>\n","    </g>\n","    <g id=\"ytick_6\">\n","     <g id=\"line2d_20\">\n","      <g>\n","       <use xlink:href=\"#mfa9bd9e3c7\" x=\"51.378125\" y=\"139.031784\" style=\"stroke: #000000; stroke-width: 0.6\"/>\n","      </g>\n","     </g>\n","    </g>\n","    <g id=\"ytick_7\">\n","     <g id=\"line2d_21\">\n","      <g>\n","       <use xlink:href=\"#mfa9bd9e3c7\" x=\"51.378125\" y=\"136.624384\" style=\"stroke: #000000; stroke-width: 0.6\"/>\n","      </g>\n","     </g>\n","    </g>\n","    <g id=\"ytick_8\">\n","     <g id=\"line2d_22\">\n","      <g>\n","       <use xlink:href=\"#mfa9bd9e3c7\" x=\"51.378125\" y=\"134.538999\" style=\"stroke: #000000; stroke-width: 0.6\"/>\n","      </g>\n","     </g>\n","    </g>\n","    <g id=\"ytick_9\">\n","     <g id=\"line2d_23\">\n","      <g>\n","       <use xlink:href=\"#mfa9bd9e3c7\" x=\"51.378125\" y=\"132.699559\" style=\"stroke: #000000; stroke-width: 0.6\"/>\n","      </g>\n","     </g>\n","    </g>\n","    <g id=\"ytick_10\">\n","     <g id=\"line2d_24\">\n","      <g>\n","       <use xlink:href=\"#mfa9bd9e3c7\" x=\"51.378125\" y=\"120.229114\" style=\"stroke: #000000; stroke-width: 0.6\"/>\n","      </g>\n","     </g>\n","    </g>\n","    <g id=\"ytick_11\">\n","     <g id=\"line2d_25\">\n","      <g>\n","       <use xlink:href=\"#mfa9bd9e3c7\" x=\"51.378125\" y=\"113.896889\" style=\"stroke: #000000; stroke-width: 0.6\"/>\n","      </g>\n","     </g>\n","    </g>\n","    <g id=\"ytick_12\">\n","     <g id=\"line2d_26\">\n","      <g>\n","       <use xlink:href=\"#mfa9bd9e3c7\" x=\"51.378125\" y=\"109.404104\" style=\"stroke: #000000; stroke-width: 0.6\"/>\n","      </g>\n","     </g>\n","    </g>\n","    <g id=\"ytick_13\">\n","     <g id=\"line2d_27\">\n","      <g>\n","       <use xlink:href=\"#mfa9bd9e3c7\" x=\"51.378125\" y=\"105.919229\" style=\"stroke: #000000; stroke-width: 0.6\"/>\n","      </g>\n","     </g>\n","    </g>\n","    <g id=\"ytick_14\">\n","     <g id=\"line2d_28\">\n","      <g>\n","       <use xlink:href=\"#mfa9bd9e3c7\" x=\"51.378125\" y=\"103.071879\" style=\"stroke: #000000; stroke-width: 0.6\"/>\n","      </g>\n","     </g>\n","    </g>\n","    <g id=\"ytick_15\">\n","     <g id=\"line2d_29\">\n","      <g>\n","       <use xlink:href=\"#mfa9bd9e3c7\" x=\"51.378125\" y=\"100.664479\" style=\"stroke: #000000; stroke-width: 0.6\"/>\n","      </g>\n","     </g>\n","    </g>\n","    <g id=\"ytick_16\">\n","     <g id=\"line2d_30\">\n","      <g>\n","       <use xlink:href=\"#mfa9bd9e3c7\" x=\"51.378125\" y=\"98.579094\" style=\"stroke: #000000; stroke-width: 0.6\"/>\n","      </g>\n","     </g>\n","    </g>\n","    <g id=\"ytick_17\">\n","     <g id=\"line2d_31\">\n","      <g>\n","       <use xlink:href=\"#mfa9bd9e3c7\" x=\"51.378125\" y=\"96.739654\" style=\"stroke: #000000; stroke-width: 0.6\"/>\n","      </g>\n","     </g>\n","    </g>\n","    <g id=\"ytick_18\">\n","     <g id=\"line2d_32\">\n","      <g>\n","       <use xlink:href=\"#mfa9bd9e3c7\" x=\"51.378125\" y=\"84.269209\" style=\"stroke: #000000; stroke-width: 0.6\"/>\n","      </g>\n","     </g>\n","    </g>\n","    <g id=\"ytick_19\">\n","     <g id=\"line2d_33\">\n","      <g>\n","       <use xlink:href=\"#mfa9bd9e3c7\" x=\"51.378125\" y=\"77.936983\" style=\"stroke: #000000; stroke-width: 0.6\"/>\n","      </g>\n","     </g>\n","    </g>\n","    <g id=\"ytick_20\">\n","     <g id=\"line2d_34\">\n","      <g>\n","       <use xlink:href=\"#mfa9bd9e3c7\" x=\"51.378125\" y=\"73.444198\" style=\"stroke: #000000; stroke-width: 0.6\"/>\n","      </g>\n","     </g>\n","    </g>\n","    <g id=\"ytick_21\">\n","     <g id=\"line2d_35\">\n","      <g>\n","       <use xlink:href=\"#mfa9bd9e3c7\" x=\"51.378125\" y=\"69.959323\" style=\"stroke: #000000; stroke-width: 0.6\"/>\n","      </g>\n","     </g>\n","    </g>\n","    <g id=\"ytick_22\">\n","     <g id=\"line2d_36\">\n","      <g>\n","       <use xlink:href=\"#mfa9bd9e3c7\" x=\"51.378125\" y=\"67.111973\" style=\"stroke: #000000; stroke-width: 0.6\"/>\n","      </g>\n","     </g>\n","    </g>\n","    <g id=\"ytick_23\">\n","     <g id=\"line2d_37\">\n","      <g>\n","       <use xlink:href=\"#mfa9bd9e3c7\" x=\"51.378125\" y=\"64.704573\" style=\"stroke: #000000; stroke-width: 0.6\"/>\n","      </g>\n","     </g>\n","    </g>\n","    <g id=\"ytick_24\">\n","     <g id=\"line2d_38\">\n","      <g>\n","       <use xlink:href=\"#mfa9bd9e3c7\" x=\"51.378125\" y=\"62.619188\" style=\"stroke: #000000; stroke-width: 0.6\"/>\n","      </g>\n","     </g>\n","    </g>\n","    <g id=\"ytick_25\">\n","     <g id=\"line2d_39\">\n","      <g>\n","       <use xlink:href=\"#mfa9bd9e3c7\" x=\"51.378125\" y=\"60.779748\" style=\"stroke: #000000; stroke-width: 0.6\"/>\n","      </g>\n","     </g>\n","    </g>\n","    <g id=\"ytick_26\">\n","     <g id=\"line2d_40\">\n","      <g>\n","       <use xlink:href=\"#mfa9bd9e3c7\" x=\"51.378125\" y=\"48.309303\" style=\"stroke: #000000; stroke-width: 0.6\"/>\n","      </g>\n","     </g>\n","    </g>\n","    <g id=\"ytick_27\">\n","     <g id=\"line2d_41\">\n","      <g>\n","       <use xlink:href=\"#mfa9bd9e3c7\" x=\"51.378125\" y=\"41.977078\" style=\"stroke: #000000; stroke-width: 0.6\"/>\n","      </g>\n","     </g>\n","    </g>\n","    <g id=\"ytick_28\">\n","     <g id=\"line2d_42\">\n","      <g>\n","       <use xlink:href=\"#mfa9bd9e3c7\" x=\"51.378125\" y=\"37.484293\" style=\"stroke: #000000; stroke-width: 0.6\"/>\n","      </g>\n","     </g>\n","    </g>\n","    <g id=\"ytick_29\">\n","     <g id=\"line2d_43\">\n","      <g>\n","       <use xlink:href=\"#mfa9bd9e3c7\" x=\"51.378125\" y=\"33.999418\" style=\"stroke: #000000; stroke-width: 0.6\"/>\n","      </g>\n","     </g>\n","    </g>\n","    <g id=\"ytick_30\">\n","     <g id=\"line2d_44\">\n","      <g>\n","       <use xlink:href=\"#mfa9bd9e3c7\" x=\"51.378125\" y=\"31.152068\" style=\"stroke: #000000; stroke-width: 0.6\"/>\n","      </g>\n","     </g>\n","    </g>\n","    <g id=\"ytick_31\">\n","     <g id=\"line2d_45\">\n","      <g>\n","       <use xlink:href=\"#mfa9bd9e3c7\" x=\"51.378125\" y=\"28.744667\" style=\"stroke: #000000; stroke-width: 0.6\"/>\n","      </g>\n","     </g>\n","    </g>\n","    <g id=\"ytick_32\">\n","     <g id=\"line2d_46\">\n","      <g>\n","       <use xlink:href=\"#mfa9bd9e3c7\" x=\"51.378125\" y=\"26.659282\" style=\"stroke: #000000; stroke-width: 0.6\"/>\n","      </g>\n","     </g>\n","    </g>\n","    <g id=\"ytick_33\">\n","     <g id=\"line2d_47\">\n","      <g>\n","       <use xlink:href=\"#mfa9bd9e3c7\" x=\"51.378125\" y=\"24.819843\" style=\"stroke: #000000; stroke-width: 0.6\"/>\n","      </g>\n","     </g>\n","    </g>\n","    <g id=\"ytick_34\">\n","     <g id=\"line2d_48\">\n","      <g>\n","       <use xlink:href=\"#mfa9bd9e3c7\" x=\"51.378125\" y=\"12.349397\" style=\"stroke: #000000; stroke-width: 0.6\"/>\n","      </g>\n","     </g>\n","    </g>\n","    <g id=\"text_11\">\n","     <!-- loss -->\n","     <g transform=\"translate(14.798437 84.807812)rotate(-90)scale(0.1 -0.1)\">\n","      <defs>\n","       <path id=\"DejaVuSans-6c\" d=\"M 603 4863 \n","L 1178 4863 \n","L 1178 0 \n","L 603 0 \n","L 603 4863 \n","z\n","\" transform=\"scale(0.015625)\"/>\n","      </defs>\n","      <use xlink:href=\"#DejaVuSans-6c\"/>\n","      <use xlink:href=\"#DejaVuSans-6f\" x=\"27.783203\"/>\n","      <use xlink:href=\"#DejaVuSans-73\" x=\"88.964844\"/>\n","      <use xlink:href=\"#DejaVuSans-73\" x=\"141.064453\"/>\n","     </g>\n","    </g>\n","   </g>\n","   <g id=\"line2d_49\">\n","    <path d=\"M 51.378125 66.609229 \n","L 61.657072 102.92158 \n","L 71.93602 124.166574 \n","L 82.214967 129.389785 \n","L 92.493914 130.171889 \n","L 102.772862 130.403184 \n","L 113.051809 130.637817 \n","L 123.330757 131.091839 \n","L 133.609704 131.591448 \n","L 143.888651 131.989853 \n","L 154.167599 132.540155 \n","L 164.446546 133.046585 \n","L 174.725493 133.574638 \n","L 185.004441 134.073878 \n","L 195.283388 134.510888 \n","L 205.562336 135.01178 \n","L 215.841283 135.521189 \n","L 226.12023 135.992137 \n","L 236.399178 136.381117 \n","L 246.678125 136.922727 \n","\" clip-path=\"url(#p97ac790d64)\" style=\"fill: none; stroke: #1f77b4; stroke-width: 1.5; stroke-linecap: square\"/>\n","   </g>\n","   <g id=\"line2d_50\">\n","    <path d=\"M 51.378125 13.377273 \n","L 61.657072 19.102244 \n","L 71.93602 24.613319 \n","L 82.214967 29.953459 \n","L 92.493914 35.111297 \n","L 102.772862 40.045542 \n","L 113.051809 44.698434 \n","L 123.330757 49.00995 \n","L 133.609704 52.909019 \n","L 143.888651 56.354886 \n","L 154.167599 59.31546 \n","L 164.446546 61.804721 \n","L 174.725493 63.873401 \n","L 185.004441 65.541152 \n","L 195.283388 66.885531 \n","L 205.562336 67.993916 \n","L 215.841283 68.902959 \n","L 226.12023 69.678381 \n","L 236.399178 70.352107 \n","L 246.678125 70.951613 \n","\" clip-path=\"url(#p97ac790d64)\" style=\"fill: none; stroke-dasharray: 5.55,2.4; stroke-dashoffset: 0; stroke: #bf00bf; stroke-width: 1.5\"/>\n","   </g>\n","   <g id=\"patch_3\">\n","    <path d=\"M 51.378125 143.1 \n","L 51.378125 7.2 \n","\" style=\"fill: none; stroke: #000000; stroke-width: 0.8; stroke-linejoin: miter; stroke-linecap: square\"/>\n","   </g>\n","   <g id=\"patch_4\">\n","    <path d=\"M 246.678125 143.1 \n","L 246.678125 7.2 \n","\" style=\"fill: none; stroke: #000000; stroke-width: 0.8; stroke-linejoin: miter; stroke-linecap: square\"/>\n","   </g>\n","   <g id=\"patch_5\">\n","    <path d=\"M 51.378125 143.1 \n","L 246.678125 143.1 \n","\" style=\"fill: none; stroke: #000000; stroke-width: 0.8; stroke-linejoin: miter; stroke-linecap: square\"/>\n","   </g>\n","   <g id=\"patch_6\">\n","    <path d=\"M 51.378125 7.2 \n","L 246.678125 7.2 \n","\" style=\"fill: none; stroke: #000000; stroke-width: 0.8; stroke-linejoin: miter; stroke-linecap: square\"/>\n","   </g>\n","   <g id=\"legend_1\">\n","    <g id=\"patch_7\">\n","     <path d=\"M 184.403125 44.55625 \n","L 239.678125 44.55625 \n","Q 241.678125 44.55625 241.678125 42.55625 \n","L 241.678125 14.2 \n","Q 241.678125 12.2 239.678125 12.2 \n","L 184.403125 12.2 \n","Q 182.403125 12.2 182.403125 14.2 \n","L 182.403125 42.55625 \n","Q 182.403125 44.55625 184.403125 44.55625 \n","z\n","\" style=\"fill: #ffffff; opacity: 0.8; stroke: #cccccc; stroke-linejoin: miter\"/>\n","    </g>\n","    <g id=\"line2d_51\">\n","     <path d=\"M 186.403125 20.298437 \n","L 196.403125 20.298437 \n","L 206.403125 20.298437 \n","\" style=\"fill: none; stroke: #1f77b4; stroke-width: 1.5; stroke-linecap: square\"/>\n","    </g>\n","    <g id=\"text_12\">\n","     <!-- train -->\n","     <g transform=\"translate(214.403125 23.798437)scale(0.1 -0.1)\">\n","      <defs>\n","       <path id=\"DejaVuSans-74\" d=\"M 1172 4494 \n","L 1172 3500 \n","L 2356 3500 \n","L 2356 3053 \n","L 1172 3053 \n","L 1172 1153 \n","Q 1172 725 1289 603 \n","Q 1406 481 1766 481 \n","L 2356 481 \n","L 2356 0 \n","L 1766 0 \n","Q 1100 0 847 248 \n","Q 594 497 594 1153 \n","L 594 3053 \n","L 172 3053 \n","L 172 3500 \n","L 594 3500 \n","L 594 4494 \n","L 1172 4494 \n","z\n","\" transform=\"scale(0.015625)\"/>\n","       <path id=\"DejaVuSans-72\" d=\"M 2631 2963 \n","Q 2534 3019 2420 3045 \n","Q 2306 3072 2169 3072 \n","Q 1681 3072 1420 2755 \n","Q 1159 2438 1159 1844 \n","L 1159 0 \n","L 581 0 \n","L 581 3500 \n","L 1159 3500 \n","L 1159 2956 \n","Q 1341 3275 1631 3429 \n","Q 1922 3584 2338 3584 \n","Q 2397 3584 2469 3576 \n","Q 2541 3569 2628 3553 \n","L 2631 2963 \n","z\n","\" transform=\"scale(0.015625)\"/>\n","       <path id=\"DejaVuSans-61\" d=\"M 2194 1759 \n","Q 1497 1759 1228 1600 \n","Q 959 1441 959 1056 \n","Q 959 750 1161 570 \n","Q 1363 391 1709 391 \n","Q 2188 391 2477 730 \n","Q 2766 1069 2766 1631 \n","L 2766 1759 \n","L 2194 1759 \n","z\n","M 3341 1997 \n","L 3341 0 \n","L 2766 0 \n","L 2766 531 \n","Q 2569 213 2275 61 \n","Q 1981 -91 1556 -91 \n","Q 1019 -91 701 211 \n","Q 384 513 384 1019 \n","Q 384 1609 779 1909 \n","Q 1175 2209 1959 2209 \n","L 2766 2209 \n","L 2766 2266 \n","Q 2766 2663 2505 2880 \n","Q 2244 3097 1772 3097 \n","Q 1472 3097 1187 3025 \n","Q 903 2953 641 2809 \n","L 641 3341 \n","Q 956 3463 1253 3523 \n","Q 1550 3584 1831 3584 \n","Q 2591 3584 2966 3190 \n","Q 3341 2797 3341 1997 \n","z\n","\" transform=\"scale(0.015625)\"/>\n","       <path id=\"DejaVuSans-69\" d=\"M 603 3500 \n","L 1178 3500 \n","L 1178 0 \n","L 603 0 \n","L 603 3500 \n","z\n","M 603 4863 \n","L 1178 4863 \n","L 1178 4134 \n","L 603 4134 \n","L 603 4863 \n","z\n","\" transform=\"scale(0.015625)\"/>\n","       <path id=\"DejaVuSans-6e\" d=\"M 3513 2113 \n","L 3513 0 \n","L 2938 0 \n","L 2938 2094 \n","Q 2938 2591 2744 2837 \n","Q 2550 3084 2163 3084 \n","Q 1697 3084 1428 2787 \n","Q 1159 2491 1159 1978 \n","L 1159 0 \n","L 581 0 \n","L 581 3500 \n","L 1159 3500 \n","L 1159 2956 \n","Q 1366 3272 1645 3428 \n","Q 1925 3584 2291 3584 \n","Q 2894 3584 3203 3211 \n","Q 3513 2838 3513 2113 \n","z\n","\" transform=\"scale(0.015625)\"/>\n","      </defs>\n","      <use xlink:href=\"#DejaVuSans-74\"/>\n","      <use xlink:href=\"#DejaVuSans-72\" x=\"39.208984\"/>\n","      <use xlink:href=\"#DejaVuSans-61\" x=\"80.322266\"/>\n","      <use xlink:href=\"#DejaVuSans-69\" x=\"141.601562\"/>\n","      <use xlink:href=\"#DejaVuSans-6e\" x=\"169.384766\"/>\n","     </g>\n","    </g>\n","    <g id=\"line2d_52\">\n","     <path d=\"M 186.403125 34.976562 \n","L 196.403125 34.976562 \n","L 206.403125 34.976562 \n","\" style=\"fill: none; stroke-dasharray: 5.55,2.4; stroke-dashoffset: 0; stroke: #bf00bf; stroke-width: 1.5\"/>\n","    </g>\n","    <g id=\"text_13\">\n","     <!-- test -->\n","     <g transform=\"translate(214.403125 38.476562)scale(0.1 -0.1)\">\n","      <use xlink:href=\"#DejaVuSans-74\"/>\n","      <use xlink:href=\"#DejaVuSans-65\" x=\"39.208984\"/>\n","      <use xlink:href=\"#DejaVuSans-73\" x=\"100.732422\"/>\n","      <use xlink:href=\"#DejaVuSans-74\" x=\"152.832031\"/>\n","     </g>\n","    </g>\n","   </g>\n","  </g>\n"," </g>\n"," <defs>\n","  <clipPath id=\"p97ac790d64\">\n","   <rect x=\"51.378125\" y=\"7.2\" width=\"195.3\" height=\"135.9\"/>\n","  </clipPath>\n"," </defs>\n","</svg>\n"],"text/plain":["<Figure size 252x180 with 1 Axes>"]},"metadata":{"needs_background":"light"},"output_type":"display_data"}],"source":["train_concise(3)"],"id":"lQO9u3UVhFj3"},{"cell_type":"markdown","metadata":{"origin_pos":32,"id":"-aicKKu9hFj3"},"source":["到目前为止，我们只接触到一个简单线性函数的概念。\n","此外，由什么构成一个简单的非线性函数可能是一个更复杂的问题。\n","例如，[再生核希尔伯特空间（RKHS）](https://en.wikipedia.org/wiki/Reproducing_kernel_Hilbert_space)\n","允许在非线性环境中应用为线性函数引入的工具。\n","不幸的是，基于RKHS的算法往往难以应用到大型、高维的数据。\n","在这本书中，我们将默认使用简单的启发式方法，即在深层网络的所有层上应用权重衰减。\n","\n","## 小结\n","\n","* 正则化是处理过拟合的常用方法：在训练集的损失函数中加入惩罚项，以降低学习到的模型的复杂度。\n","* 保持模型简单的一个特别的选择是使用$L_2$惩罚的权重衰减。这会导致学习算法更新步骤中的权重衰减。\n","* 权重衰减功能在深度学习框架的优化器中提供。\n","* 在同一训练代码实现中，不同的参数集可以有不同的更新行为。\n","\n","## 练习\n","\n","1. 在本节的估计问题中使用$\\lambda$的值进行实验。绘制训练和测试精度关于$\\lambda$的函数。你观察到了什么？\n","1. 使用验证集来找到最佳值$\\lambda$。它真的是最优值吗？这有关系吗？\n","1. 如果我们使用$\\sum_i |w_i|$作为我们选择的惩罚（$L_1$正则化），那么更新方程会是什么样子？\n","1. 我们知道$\\|\\mathbf{w}\\|^2 = \\mathbf{w}^\\top \\mathbf{w}$。你能找到类似的矩阵方程吗（见 :numref:`subsec_lin-algebra-norms` 中的Frobenius范数）？\n","1. 回顾训练误差和泛化误差之间的关系。除了权重衰减、增加训练数据、使用适当复杂度的模型之外，你还能想出其他什么方法来处理过拟合？\n","1. 在贝叶斯统计中，我们使用先验和似然的乘积，通过公式$P(w \\mid x) \\propto P(x \\mid w) P(w)$得到后验。如何得到带正则化的$P(w)$？\n"],"id":"-aicKKu9hFj3"},{"cell_type":"markdown","metadata":{"origin_pos":0,"id":"a5baedd2"},"source":["# 暂退法（Dropout）\n",":label:`sec_dropout`\n","\n","在 :numref:`sec_weight_decay` 中，\n","我们介绍了通过惩罚权重的$L_2$范数来正则化统计模型的经典方法。\n","在概率角度看，我们可以通过以下论证来证明这一技术的合理性：\n","我们已经假设了一个先验，即权重的值取自均值为0的高斯分布。\n","更直观的是，我们希望模型深度挖掘特征，即将其权重分散到许多特征中，\n","而不是过于依赖少数潜在的虚假关联。\n","\n","## 重新审视过拟合\n","\n","当面对更多的特征而样本不足时，线性模型往往会过拟合。\n","相反，当给出更多样本而不是特征，通常线性模型不会过拟合。\n","不幸的是，线性模型泛化的可靠性是有代价的。\n","简单地说，线性模型没有考虑到特征之间的交互作用。\n","对于每个特征，线性模型必须指定正的或负的权重，而忽略其他特征。\n","\n","泛化性和灵活性之间的这种基本权衡被描述为*偏差-方差权衡*（bias-variance tradeoff）。\n","线性模型有很高的偏差：它们只能表示一小类函数。\n","然而，这些模型的方差很低：它们在不同的随机数据样本上可以得出相似的结果。\n","\n","深度神经网络位于偏差-方差谱的另一端。\n","与线性模型不同，神经网络并不局限于单独查看每个特征，而是学习特征之间的交互。\n","例如，神经网络可能推断“尼日利亚”和“西联汇款”一起出现在电子邮件中表示垃圾邮件，\n","但单独出现则不表示垃圾邮件。\n","\n","即使我们有比特征多得多的样本，深度神经网络也有可能过拟合。\n","2017年，一组研究人员通过在随机标记的图像上训练深度网络。\n","这展示了神经网络的极大灵活性，因为人类很难将输入和随机标记的输出联系起来，\n","但通过随机梯度下降优化的神经网络可以完美地标记训练集中的每一幅图像。\n","想一想这意味着什么？\n","假设标签是随机均匀分配的，并且有10个类别，那么分类器在测试数据上很难取得高于10%的精度，\n","那么这里的泛化差距就高达90%，如此严重的过拟合。\n","\n","深度网络的泛化性质令人费解，而这种泛化性质的数学基础仍然是悬而未决的研究问题。\n","我们鼓励喜好研究理论的读者更深入地研究这个主题。\n","本节，我们将着重对实际工具的探究，这些工具倾向于改进深层网络的泛化性。\n","\n","## 扰动的稳健性\n","\n","在探究泛化性之前，我们先来定义一下什么是一个“好”的预测模型？\n","我们期待“好”的预测模型能在未知的数据上有很好的表现：\n","经典泛化理论认为，为了缩小训练和测试性能之间的差距，应该以简单的模型为目标。\n","简单性以较小维度的形式展现，\n","我们在 :numref:`sec_model_selection` 讨论线性模型的单项式函数时探讨了这一点。\n","此外，正如我们在 :numref:`sec_weight_decay` 中讨论权重衰减（$L_2$正则化）时看到的那样，\n","参数的范数也代表了一种有用的简单性度量。\n","\n","简单性的另一个角度是平滑性，即函数不应该对其输入的微小变化敏感。\n","例如，当我们对图像进行分类时，我们预计向像素添加一些随机噪声应该是基本无影响的。\n","1995年，克里斯托弗·毕晓普证明了\n","具有输入噪声的训练等价于Tikhonov正则化 :cite:`Bishop.1995`。\n","这项工作用数学证实了“要求函数光滑”和“要求函数对输入的随机噪声具有适应性”之间的联系。\n","\n","然后在2014年，斯里瓦斯塔瓦等人 :cite:`Srivastava.Hinton.Krizhevsky.ea.2014`\n","就如何将毕晓普的想法应用于网络的内部层提出了一个想法：\n","在训练过程中，他们建议在计算后续层之前向网络的每一层注入噪声。\n","因为当训练一个有多层的深层网络时，注入噪声只会在输入-输出映射上增强平滑性。\n","\n","这个想法被称为*暂退法*（dropout）。\n","暂退法在前向传播过程中，计算每一内部层的同时注入噪声，这已经成为训练神经网络的常用技术。\n","这种方法之所以被称为*暂退法*，因为我们从表面上看是在训练过程中丢弃（drop out）一些神经元。\n","在整个训练过程的每一次迭代中，标准暂退法包括在计算下一层之前将当前层中的一些节点置零。\n","\n","需要说明的是，暂退法的原始论文提到了一个关于有性繁殖的类比：\n","神经网络过拟合与每一层都依赖于前一层激活值相关，称这种情况为“共适应性”。\n","作者认为，暂退法会破坏共适应性，就像有性生殖会破坏共适应的基因一样。\n","\n","那么关键的挑战就是如何注入这种噪声。\n","一种想法是以一种*无偏向*（unbiased）的方式注入噪声。\n","这样在固定住其他层时，每一层的期望值等于没有噪音时的值。\n","\n","在毕晓普的工作中，他将高斯噪声添加到线性模型的输入中。\n","在每次训练迭代中，他将从均值为零的分布$\\epsilon \\sim \\mathcal{N}(0,\\sigma^2)$\n","采样噪声添加到输入$\\mathbf{x}$，\n","从而产生扰动点$\\mathbf{x}' = \\mathbf{x} + \\epsilon$，\n","预期是$E[\\mathbf{x}'] = \\mathbf{x}$。\n","\n","在标准暂退法正则化中，通过按保留（未丢弃）的节点的分数进行规范化来消除每一层的偏差。\n","换言之，每个中间活性值$h$以*暂退概率*$p$由随机变量$h'$替换，如下所示：\n","\n","$$\n","\\begin{aligned}\n","h' =\n","\\begin{cases}\n","    0 & \\text{ 概率为 } p \\\\\n","    \\frac{h}{1-p} & \\text{ 其他情况}\n","\\end{cases}\n","\\end{aligned}\n","$$\n","\n","根据此模型的设计，其期望值保持不变，即$E[h'] = h$。\n","\n","## 实践中的暂退法\n","\n","回想一下 :numref:`fig_mlp`中带有1个隐藏层和5个隐藏单元的多层感知机。\n","当我们将暂退法应用到隐藏层，以$p$的概率将隐藏单元置为零时，\n","结果可以看作是一个只包含原始神经元子集的网络。\n","比如在 :numref:`fig_dropout2`中，删除了$h_2$和$h_5$，\n","因此输出的计算不再依赖于$h_2$或$h_5$，并且它们各自的梯度在执行反向传播时也会消失。\n","这样，输出层的计算不能过度依赖于$h_1, \\ldots, h_5$的任何一个元素。\n","\n","![dropout前后的多层感知机](http://d2l.ai/_images/dropout2.svg)\n",":label:`fig_dropout2`\n","\n","通常，我们在测试时不用暂退法。\n","给定一个训练好的模型和一个新的样本，我们不会丢弃任何节点，因此不需要标准化。\n","然而也有一些例外：一些研究人员在测试时使用暂退法，\n","用于估计神经网络预测的“不确定性”：\n","如果通过许多不同的暂退法遮盖后得到的预测结果都是一致的，那么我们可以说网络发挥更稳定。\n","\n","## 从零开始实现\n","\n","要实现单层的暂退法函数，\n","我们从均匀分布$U[0, 1]$中抽取样本，样本数与这层神经网络的维度一致。\n","然后我们保留那些对应样本大于$p$的节点，把剩下的丢弃。\n","\n","在下面的代码中，(**我们实现 `dropout_layer` 函数，\n","该函数以`dropout`的概率丢弃张量输入`X`中的元素**)，\n","如上所述重新缩放剩余部分：将剩余部分除以`1.0-dropout`。\n"],"id":"a5baedd2"},{"cell_type":"code","execution_count":null,"metadata":{"execution":{"iopub.execute_input":"2022-07-31T02:18:37.320831Z","iopub.status.busy":"2022-07-31T02:18:37.320249Z","iopub.status.idle":"2022-07-31T02:18:39.269480Z","shell.execute_reply":"2022-07-31T02:18:39.268656Z"},"origin_pos":2,"tab":["pytorch"],"id":"16b50c90"},"outputs":[],"source":["import torch\n","from torch import nn\n","from d2l import torch as d2l\n","\n","\n","def dropout_layer(X, dropout):\n","    assert 0 <= dropout <= 1\n","    # 在本情况中，所有元素都被丢弃\n","    if dropout == 1:\n","        return torch.zeros_like(X)\n","    # 在本情况中，所有元素都被保留\n","    if dropout == 0:\n","        return X\n","    mask = (torch.rand(X.shape) > dropout).float()\n","    return mask * X / (1.0 - dropout)"],"id":"16b50c90"},{"cell_type":"markdown","metadata":{"origin_pos":4,"id":"83f6d1b0"},"source":["我们可以通过下面几个例子来[**测试`dropout_layer`函数**]。\n","我们将输入`X`通过暂退法操作，暂退概率分别为0、0.5和1。\n"],"id":"83f6d1b0"},{"cell_type":"code","execution_count":null,"metadata":{"execution":{"iopub.execute_input":"2022-07-31T02:18:39.273487Z","iopub.status.busy":"2022-07-31T02:18:39.272952Z","iopub.status.idle":"2022-07-31T02:18:39.280148Z","shell.execute_reply":"2022-07-31T02:18:39.279502Z"},"origin_pos":6,"tab":["pytorch"],"id":"60923d0c","outputId":"4655928a-15aa-489e-bf8f-d48da14424ee"},"outputs":[{"name":"stdout","output_type":"stream","text":["tensor([[ 0.,  1.,  2.,  3.,  4.,  5.,  6.,  7.],\n","        [ 8.,  9., 10., 11., 12., 13., 14., 15.]])\n","tensor([[ 0.,  1.,  2.,  3.,  4.,  5.,  6.,  7.],\n","        [ 8.,  9., 10., 11., 12., 13., 14., 15.]])\n","tensor([[ 0.,  0.,  0.,  0.,  8.,  0.,  0., 14.],\n","        [16.,  0.,  0.,  0., 24.,  0.,  0.,  0.]])\n","tensor([[0., 0., 0., 0., 0., 0., 0., 0.],\n","        [0., 0., 0., 0., 0., 0., 0., 0.]])\n"]}],"source":["X= torch.arange(16, dtype = torch.float32).reshape((2, 8))\n","print(X)\n","print(dropout_layer(X, 0.))\n","print(dropout_layer(X, 0.5))\n","print(dropout_layer(X, 1.))"],"id":"60923d0c"},{"cell_type":"markdown","metadata":{"origin_pos":8,"id":"3421b121"},"source":["### 定义模型参数\n","\n","同样，我们使用 :numref:`sec_fashion_mnist`中引入的Fashion-MNIST数据集。\n","我们[**定义具有两个隐藏层的多层感知机，每个隐藏层包含256个单元**]。\n"],"id":"3421b121"},{"cell_type":"code","execution_count":null,"metadata":{"execution":{"iopub.execute_input":"2022-07-31T02:18:39.283459Z","iopub.status.busy":"2022-07-31T02:18:39.282880Z","iopub.status.idle":"2022-07-31T02:18:39.286706Z","shell.execute_reply":"2022-07-31T02:18:39.286027Z"},"origin_pos":10,"tab":["pytorch"],"id":"87e5b031"},"outputs":[],"source":["num_inputs, num_outputs, num_hiddens1, num_hiddens2 = 784, 10, 256, 256"],"id":"87e5b031"},{"cell_type":"markdown","metadata":{"origin_pos":12,"id":"092fe2d8"},"source":["### 定义模型\n","\n","我们可以将暂退法应用于每个隐藏层的输出（在激活函数之后），\n","并且可以为每一层分别设置暂退概率：\n","常见的技巧是在靠近输入层的地方设置较低的暂退概率。\n","下面的模型将第一个和第二个隐藏层的暂退概率分别设置为0.2和0.5，\n","并且暂退法只在训练期间有效。\n"],"id":"092fe2d8"},{"cell_type":"code","execution_count":null,"metadata":{"execution":{"iopub.execute_input":"2022-07-31T02:18:39.289625Z","iopub.status.busy":"2022-07-31T02:18:39.289310Z","iopub.status.idle":"2022-07-31T02:18:39.299676Z","shell.execute_reply":"2022-07-31T02:18:39.298984Z"},"origin_pos":14,"tab":["pytorch"],"id":"59e29f33"},"outputs":[],"source":["dropout1, dropout2 = 0.2, 0.5\n","\n","class Net(nn.Module):\n","    def __init__(self, num_inputs, num_outputs, num_hiddens1, num_hiddens2,\n","                 is_training = True):\n","        super(Net, self).__init__()\n","        self.num_inputs = num_inputs\n","        self.training = is_training\n","        self.lin1 = nn.Linear(num_inputs, num_hiddens1)\n","        self.lin2 = nn.Linear(num_hiddens1, num_hiddens2)\n","        self.lin3 = nn.Linear(num_hiddens2, num_outputs)\n","        self.relu = nn.ReLU()\n","\n","    def forward(self, X):\n","        H1 = self.relu(self.lin1(X.reshape((-1, self.num_inputs))))\n","        # 只有在训练模型时才使用dropout\n","        if self.training == True:\n","            # 在第一个全连接层之后添加一个dropout层\n","            H1 = dropout_layer(H1, dropout1)\n","        H2 = self.relu(self.lin2(H1))\n","        if self.training == True:\n","            # 在第二个全连接层之后添加一个dropout层\n","            H2 = dropout_layer(H2, dropout2)\n","        out = self.lin3(H2)\n","        return out\n","\n","\n","net = Net(num_inputs, num_outputs, num_hiddens1, num_hiddens2)"],"id":"59e29f33"},{"cell_type":"markdown","metadata":{"origin_pos":16,"id":"45ccf0a2"},"source":["### [**训练和测试**]\n","\n","这类似于前面描述的多层感知机训练和测试。\n"],"id":"45ccf0a2"},{"cell_type":"code","execution_count":null,"metadata":{"execution":{"iopub.execute_input":"2022-07-31T02:18:39.302897Z","iopub.status.busy":"2022-07-31T02:18:39.302269Z","iopub.status.idle":"2022-07-31T02:19:44.136602Z","shell.execute_reply":"2022-07-31T02:19:44.135572Z"},"origin_pos":18,"tab":["pytorch"],"id":"74f2a8aa","outputId":"16c065ff-7877-405b-ff03-c1996027613d"},"outputs":[{"data":{"image/svg+xml":["<?xml version=\"1.0\" encoding=\"utf-8\" standalone=\"no\"?>\n","<!DOCTYPE svg PUBLIC \"-//W3C//DTD SVG 1.1//EN\"\n","  \"http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd\">\n","<svg xmlns:xlink=\"http://www.w3.org/1999/xlink\" width=\"238.965625pt\" height=\"180.65625pt\" viewBox=\"0 0 238.965625 180.65625\" xmlns=\"http://www.w3.org/2000/svg\" version=\"1.1\">\n"," <metadata>\n","  <rdf:RDF xmlns:dc=\"http://purl.org/dc/elements/1.1/\" xmlns:cc=\"http://creativecommons.org/ns#\" xmlns:rdf=\"http://www.w3.org/1999/02/22-rdf-syntax-ns#\">\n","   <cc:Work>\n","    <dc:type rdf:resource=\"http://purl.org/dc/dcmitype/StillImage\"/>\n","    <dc:date>2022-07-31T02:19:44.095570</dc:date>\n","    <dc:format>image/svg+xml</dc:format>\n","    <dc:creator>\n","     <cc:Agent>\n","      <dc:title>Matplotlib v3.5.1, https://matplotlib.org/</dc:title>\n","     </cc:Agent>\n","    </dc:creator>\n","   </cc:Work>\n","  </rdf:RDF>\n"," </metadata>\n"," <defs>\n","  <style type=\"text/css\">*{stroke-linejoin: round; stroke-linecap: butt}</style>\n"," </defs>\n"," <g id=\"figure_1\">\n","  <g id=\"patch_1\">\n","   <path d=\"M 0 180.65625 \n","L 238.965625 180.65625 \n","L 238.965625 0 \n","L 0 0 \n","L 0 180.65625 \n","z\n","\" style=\"fill: none\"/>\n","  </g>\n","  <g id=\"axes_1\">\n","   <g id=\"patch_2\">\n","    <path d=\"M 30.103125 143.1 \n","L 225.403125 143.1 \n","L 225.403125 7.2 \n","L 30.103125 7.2 \n","z\n","\" style=\"fill: #ffffff\"/>\n","   </g>\n","   <g id=\"matplotlib.axis_1\">\n","    <g id=\"xtick_1\">\n","     <g id=\"line2d_1\">\n","      <path d=\"M 51.803125 143.1 \n","L 51.803125 7.2 \n","\" clip-path=\"url(#p85b7155709)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n","     </g>\n","     <g id=\"line2d_2\">\n","      <defs>\n","       <path id=\"mb990d9cad0\" d=\"M 0 0 \n","L 0 3.5 \n","\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n","      </defs>\n","      <g>\n","       <use xlink:href=\"#mb990d9cad0\" x=\"51.803125\" y=\"143.1\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n","      </g>\n","     </g>\n","     <g id=\"text_1\">\n","      <!-- 2 -->\n","      <g transform=\"translate(48.621875 157.698438)scale(0.1 -0.1)\">\n","       <defs>\n","        <path id=\"DejaVuSans-32\" d=\"M 1228 531 \n","L 3431 531 \n","L 3431 0 \n","L 469 0 \n","L 469 531 \n","Q 828 903 1448 1529 \n","Q 2069 2156 2228 2338 \n","Q 2531 2678 2651 2914 \n","Q 2772 3150 2772 3378 \n","Q 2772 3750 2511 3984 \n","Q 2250 4219 1831 4219 \n","Q 1534 4219 1204 4116 \n","Q 875 4013 500 3803 \n","L 500 4441 \n","Q 881 4594 1212 4672 \n","Q 1544 4750 1819 4750 \n","Q 2544 4750 2975 4387 \n","Q 3406 4025 3406 3419 \n","Q 3406 3131 3298 2873 \n","Q 3191 2616 2906 2266 \n","Q 2828 2175 2409 1742 \n","Q 1991 1309 1228 531 \n","z\n","\" transform=\"scale(0.015625)\"/>\n","       </defs>\n","       <use xlink:href=\"#DejaVuSans-32\"/>\n","      </g>\n","     </g>\n","    </g>\n","    <g id=\"xtick_2\">\n","     <g id=\"line2d_3\">\n","      <path d=\"M 95.203125 143.1 \n","L 95.203125 7.2 \n","\" clip-path=\"url(#p85b7155709)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n","     </g>\n","     <g id=\"line2d_4\">\n","      <g>\n","       <use xlink:href=\"#mb990d9cad0\" x=\"95.203125\" y=\"143.1\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n","      </g>\n","     </g>\n","     <g id=\"text_2\">\n","      <!-- 4 -->\n","      <g transform=\"translate(92.021875 157.698438)scale(0.1 -0.1)\">\n","       <defs>\n","        <path id=\"DejaVuSans-34\" d=\"M 2419 4116 \n","L 825 1625 \n","L 2419 1625 \n","L 2419 4116 \n","z\n","M 2253 4666 \n","L 3047 4666 \n","L 3047 1625 \n","L 3713 1625 \n","L 3713 1100 \n","L 3047 1100 \n","L 3047 0 \n","L 2419 0 \n","L 2419 1100 \n","L 313 1100 \n","L 313 1709 \n","L 2253 4666 \n","z\n","\" transform=\"scale(0.015625)\"/>\n","       </defs>\n","       <use xlink:href=\"#DejaVuSans-34\"/>\n","      </g>\n","     </g>\n","    </g>\n","    <g id=\"xtick_3\">\n","     <g id=\"line2d_5\">\n","      <path d=\"M 138.603125 143.1 \n","L 138.603125 7.2 \n","\" clip-path=\"url(#p85b7155709)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n","     </g>\n","     <g id=\"line2d_6\">\n","      <g>\n","       <use xlink:href=\"#mb990d9cad0\" x=\"138.603125\" y=\"143.1\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n","      </g>\n","     </g>\n","     <g id=\"text_3\">\n","      <!-- 6 -->\n","      <g transform=\"translate(135.421875 157.698438)scale(0.1 -0.1)\">\n","       <defs>\n","        <path id=\"DejaVuSans-36\" d=\"M 2113 2584 \n","Q 1688 2584 1439 2293 \n","Q 1191 2003 1191 1497 \n","Q 1191 994 1439 701 \n","Q 1688 409 2113 409 \n","Q 2538 409 2786 701 \n","Q 3034 994 3034 1497 \n","Q 3034 2003 2786 2293 \n","Q 2538 2584 2113 2584 \n","z\n","M 3366 4563 \n","L 3366 3988 \n","Q 3128 4100 2886 4159 \n","Q 2644 4219 2406 4219 \n","Q 1781 4219 1451 3797 \n","Q 1122 3375 1075 2522 \n","Q 1259 2794 1537 2939 \n","Q 1816 3084 2150 3084 \n","Q 2853 3084 3261 2657 \n","Q 3669 2231 3669 1497 \n","Q 3669 778 3244 343 \n","Q 2819 -91 2113 -91 \n","Q 1303 -91 875 529 \n","Q 447 1150 447 2328 \n","Q 447 3434 972 4092 \n","Q 1497 4750 2381 4750 \n","Q 2619 4750 2861 4703 \n","Q 3103 4656 3366 4563 \n","z\n","\" transform=\"scale(0.015625)\"/>\n","       </defs>\n","       <use xlink:href=\"#DejaVuSans-36\"/>\n","      </g>\n","     </g>\n","    </g>\n","    <g id=\"xtick_4\">\n","     <g id=\"line2d_7\">\n","      <path d=\"M 182.003125 143.1 \n","L 182.003125 7.2 \n","\" clip-path=\"url(#p85b7155709)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n","     </g>\n","     <g id=\"line2d_8\">\n","      <g>\n","       <use xlink:href=\"#mb990d9cad0\" x=\"182.003125\" y=\"143.1\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n","      </g>\n","     </g>\n","     <g id=\"text_4\">\n","      <!-- 8 -->\n","      <g transform=\"translate(178.821875 157.698438)scale(0.1 -0.1)\">\n","       <defs>\n","        <path id=\"DejaVuSans-38\" d=\"M 2034 2216 \n","Q 1584 2216 1326 1975 \n","Q 1069 1734 1069 1313 \n","Q 1069 891 1326 650 \n","Q 1584 409 2034 409 \n","Q 2484 409 2743 651 \n","Q 3003 894 3003 1313 \n","Q 3003 1734 2745 1975 \n","Q 2488 2216 2034 2216 \n","z\n","M 1403 2484 \n","Q 997 2584 770 2862 \n","Q 544 3141 544 3541 \n","Q 544 4100 942 4425 \n","Q 1341 4750 2034 4750 \n","Q 2731 4750 3128 4425 \n","Q 3525 4100 3525 3541 \n","Q 3525 3141 3298 2862 \n","Q 3072 2584 2669 2484 \n","Q 3125 2378 3379 2068 \n","Q 3634 1759 3634 1313 \n","Q 3634 634 3220 271 \n","Q 2806 -91 2034 -91 \n","Q 1263 -91 848 271 \n","Q 434 634 434 1313 \n","Q 434 1759 690 2068 \n","Q 947 2378 1403 2484 \n","z\n","M 1172 3481 \n","Q 1172 3119 1398 2916 \n","Q 1625 2713 2034 2713 \n","Q 2441 2713 2670 2916 \n","Q 2900 3119 2900 3481 \n","Q 2900 3844 2670 4047 \n","Q 2441 4250 2034 4250 \n","Q 1625 4250 1398 4047 \n","Q 1172 3844 1172 3481 \n","z\n","\" transform=\"scale(0.015625)\"/>\n","       </defs>\n","       <use xlink:href=\"#DejaVuSans-38\"/>\n","      </g>\n","     </g>\n","    </g>\n","    <g id=\"xtick_5\">\n","     <g id=\"line2d_9\">\n","      <path d=\"M 225.403125 143.1 \n","L 225.403125 7.2 \n","\" clip-path=\"url(#p85b7155709)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n","     </g>\n","     <g id=\"line2d_10\">\n","      <g>\n","       <use xlink:href=\"#mb990d9cad0\" x=\"225.403125\" y=\"143.1\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n","      </g>\n","     </g>\n","     <g id=\"text_5\">\n","      <!-- 10 -->\n","      <g transform=\"translate(219.040625 157.698438)scale(0.1 -0.1)\">\n","       <defs>\n","        <path id=\"DejaVuSans-31\" d=\"M 794 531 \n","L 1825 531 \n","L 1825 4091 \n","L 703 3866 \n","L 703 4441 \n","L 1819 4666 \n","L 2450 4666 \n","L 2450 531 \n","L 3481 531 \n","L 3481 0 \n","L 794 0 \n","L 794 531 \n","z\n","\" transform=\"scale(0.015625)\"/>\n","        <path id=\"DejaVuSans-30\" d=\"M 2034 4250 \n","Q 1547 4250 1301 3770 \n","Q 1056 3291 1056 2328 \n","Q 1056 1369 1301 889 \n","Q 1547 409 2034 409 \n","Q 2525 409 2770 889 \n","Q 3016 1369 3016 2328 \n","Q 3016 3291 2770 3770 \n","Q 2525 4250 2034 4250 \n","z\n","M 2034 4750 \n","Q 2819 4750 3233 4129 \n","Q 3647 3509 3647 2328 \n","Q 3647 1150 3233 529 \n","Q 2819 -91 2034 -91 \n","Q 1250 -91 836 529 \n","Q 422 1150 422 2328 \n","Q 422 3509 836 4129 \n","Q 1250 4750 2034 4750 \n","z\n","\" transform=\"scale(0.015625)\"/>\n","       </defs>\n","       <use xlink:href=\"#DejaVuSans-31\"/>\n","       <use xlink:href=\"#DejaVuSans-30\" x=\"63.623047\"/>\n","      </g>\n","     </g>\n","    </g>\n","    <g id=\"text_6\">\n","     <!-- epoch -->\n","     <g transform=\"translate(112.525 171.376563)scale(0.1 -0.1)\">\n","      <defs>\n","       <path id=\"DejaVuSans-65\" d=\"M 3597 1894 \n","L 3597 1613 \n","L 953 1613 \n","Q 991 1019 1311 708 \n","Q 1631 397 2203 397 \n","Q 2534 397 2845 478 \n","Q 3156 559 3463 722 \n","L 3463 178 \n","Q 3153 47 2828 -22 \n","Q 2503 -91 2169 -91 \n","Q 1331 -91 842 396 \n","Q 353 884 353 1716 \n","Q 353 2575 817 3079 \n","Q 1281 3584 2069 3584 \n","Q 2775 3584 3186 3129 \n","Q 3597 2675 3597 1894 \n","z\n","M 3022 2063 \n","Q 3016 2534 2758 2815 \n","Q 2500 3097 2075 3097 \n","Q 1594 3097 1305 2825 \n","Q 1016 2553 972 2059 \n","L 3022 2063 \n","z\n","\" transform=\"scale(0.015625)\"/>\n","       <path id=\"DejaVuSans-70\" d=\"M 1159 525 \n","L 1159 -1331 \n","L 581 -1331 \n","L 581 3500 \n","L 1159 3500 \n","L 1159 2969 \n","Q 1341 3281 1617 3432 \n","Q 1894 3584 2278 3584 \n","Q 2916 3584 3314 3078 \n","Q 3713 2572 3713 1747 \n","Q 3713 922 3314 415 \n","Q 2916 -91 2278 -91 \n","Q 1894 -91 1617 61 \n","Q 1341 213 1159 525 \n","z\n","M 3116 1747 \n","Q 3116 2381 2855 2742 \n","Q 2594 3103 2138 3103 \n","Q 1681 3103 1420 2742 \n","Q 1159 2381 1159 1747 \n","Q 1159 1113 1420 752 \n","Q 1681 391 2138 391 \n","Q 2594 391 2855 752 \n","Q 3116 1113 3116 1747 \n","z\n","\" transform=\"scale(0.015625)\"/>\n","       <path id=\"DejaVuSans-6f\" d=\"M 1959 3097 \n","Q 1497 3097 1228 2736 \n","Q 959 2375 959 1747 \n","Q 959 1119 1226 758 \n","Q 1494 397 1959 397 \n","Q 2419 397 2687 759 \n","Q 2956 1122 2956 1747 \n","Q 2956 2369 2687 2733 \n","Q 2419 3097 1959 3097 \n","z\n","M 1959 3584 \n","Q 2709 3584 3137 3096 \n","Q 3566 2609 3566 1747 \n","Q 3566 888 3137 398 \n","Q 2709 -91 1959 -91 \n","Q 1206 -91 779 398 \n","Q 353 888 353 1747 \n","Q 353 2609 779 3096 \n","Q 1206 3584 1959 3584 \n","z\n","\" transform=\"scale(0.015625)\"/>\n","       <path id=\"DejaVuSans-63\" d=\"M 3122 3366 \n","L 3122 2828 \n","Q 2878 2963 2633 3030 \n","Q 2388 3097 2138 3097 \n","Q 1578 3097 1268 2742 \n","Q 959 2388 959 1747 \n","Q 959 1106 1268 751 \n","Q 1578 397 2138 397 \n","Q 2388 397 2633 464 \n","Q 2878 531 3122 666 \n","L 3122 134 \n","Q 2881 22 2623 -34 \n","Q 2366 -91 2075 -91 \n","Q 1284 -91 818 406 \n","Q 353 903 353 1747 \n","Q 353 2603 823 3093 \n","Q 1294 3584 2113 3584 \n","Q 2378 3584 2631 3529 \n","Q 2884 3475 3122 3366 \n","z\n","\" transform=\"scale(0.015625)\"/>\n","       <path id=\"DejaVuSans-68\" d=\"M 3513 2113 \n","L 3513 0 \n","L 2938 0 \n","L 2938 2094 \n","Q 2938 2591 2744 2837 \n","Q 2550 3084 2163 3084 \n","Q 1697 3084 1428 2787 \n","Q 1159 2491 1159 1978 \n","L 1159 0 \n","L 581 0 \n","L 581 4863 \n","L 1159 4863 \n","L 1159 2956 \n","Q 1366 3272 1645 3428 \n","Q 1925 3584 2291 3584 \n","Q 2894 3584 3203 3211 \n","Q 3513 2838 3513 2113 \n","z\n","\" transform=\"scale(0.015625)\"/>\n","      </defs>\n","      <use xlink:href=\"#DejaVuSans-65\"/>\n","      <use xlink:href=\"#DejaVuSans-70\" x=\"61.523438\"/>\n","      <use xlink:href=\"#DejaVuSans-6f\" x=\"125\"/>\n","      <use xlink:href=\"#DejaVuSans-63\" x=\"186.181641\"/>\n","      <use xlink:href=\"#DejaVuSans-68\" x=\"241.162109\"/>\n","     </g>\n","    </g>\n","   </g>\n","   <g id=\"matplotlib.axis_2\">\n","    <g id=\"ytick_1\">\n","     <g id=\"line2d_11\">\n","      <path d=\"M 30.103125 120.45 \n","L 225.403125 120.45 \n","\" clip-path=\"url(#p85b7155709)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n","     </g>\n","     <g id=\"line2d_12\">\n","      <defs>\n","       <path id=\"md2ee03d612\" d=\"M 0 0 \n","L -3.5 0 \n","\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n","      </defs>\n","      <g>\n","       <use xlink:href=\"#md2ee03d612\" x=\"30.103125\" y=\"120.45\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n","      </g>\n","     </g>\n","     <g id=\"text_7\">\n","      <!-- 0.4 -->\n","      <g transform=\"translate(7.2 124.249219)scale(0.1 -0.1)\">\n","       <defs>\n","        <path id=\"DejaVuSans-2e\" d=\"M 684 794 \n","L 1344 794 \n","L 1344 0 \n","L 684 0 \n","L 684 794 \n","z\n","\" transform=\"scale(0.015625)\"/>\n","       </defs>\n","       <use xlink:href=\"#DejaVuSans-30\"/>\n","       <use xlink:href=\"#DejaVuSans-2e\" x=\"63.623047\"/>\n","       <use xlink:href=\"#DejaVuSans-34\" x=\"95.410156\"/>\n","      </g>\n","     </g>\n","    </g>\n","    <g id=\"ytick_2\">\n","     <g id=\"line2d_13\">\n","      <path d=\"M 30.103125 75.15 \n","L 225.403125 75.15 \n","\" clip-path=\"url(#p85b7155709)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n","     </g>\n","     <g id=\"line2d_14\">\n","      <g>\n","       <use xlink:href=\"#md2ee03d612\" x=\"30.103125\" y=\"75.15\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n","      </g>\n","     </g>\n","     <g id=\"text_8\">\n","      <!-- 0.6 -->\n","      <g transform=\"translate(7.2 78.949219)scale(0.1 -0.1)\">\n","       <use xlink:href=\"#DejaVuSans-30\"/>\n","       <use xlink:href=\"#DejaVuSans-2e\" x=\"63.623047\"/>\n","       <use xlink:href=\"#DejaVuSans-36\" x=\"95.410156\"/>\n","      </g>\n","     </g>\n","    </g>\n","    <g id=\"ytick_3\">\n","     <g id=\"line2d_15\">\n","      <path d=\"M 30.103125 29.85 \n","L 225.403125 29.85 \n","\" clip-path=\"url(#p85b7155709)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n","     </g>\n","     <g id=\"line2d_16\">\n","      <g>\n","       <use xlink:href=\"#md2ee03d612\" x=\"30.103125\" y=\"29.85\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n","      </g>\n","     </g>\n","     <g id=\"text_9\">\n","      <!-- 0.8 -->\n","      <g transform=\"translate(7.2 33.649219)scale(0.1 -0.1)\">\n","       <use xlink:href=\"#DejaVuSans-30\"/>\n","       <use xlink:href=\"#DejaVuSans-2e\" x=\"63.623047\"/>\n","       <use xlink:href=\"#DejaVuSans-38\" x=\"95.410156\"/>\n","      </g>\n","     </g>\n","    </g>\n","   </g>\n","   <g id=\"line2d_17\">\n","    <path d=\"M 30.103125 12.324623 \n","L 51.803125 90.927617 \n","L 73.503125 106.365164 \n","L 95.203125 114.875577 \n","L 116.903125 119.976358 \n","L 138.603125 124.696432 \n","L 160.303125 127.613197 \n","L 182.003125 129.904184 \n","L 203.703125 132.672218 \n","L 225.403125 134.818051 \n","\" clip-path=\"url(#p85b7155709)\" style=\"fill: none; stroke: #1f77b4; stroke-width: 1.5; stroke-linecap: square\"/>\n","   </g>\n","   <g id=\"line2d_18\">\n","    <path d=\"M 30.103125 58.9326 \n","L 51.803125 28.8232 \n","L 73.503125 22.907775 \n","L 95.203125 19.64995 \n","L 116.903125 17.713375 \n","L 138.603125 16.052375 \n","L 160.303125 15.184125 \n","L 182.003125 14.24415 \n","L 203.703125 13.447625 \n","L 225.403125 12.8021 \n","\" clip-path=\"url(#p85b7155709)\" style=\"fill: none; stroke-dasharray: 5.55,2.4; stroke-dashoffset: 0; stroke: #bf00bf; stroke-width: 1.5\"/>\n","   </g>\n","   <g id=\"line2d_19\">\n","    <path d=\"M 30.103125 30.0765 \n","L 51.803125 36.16935 \n","L 73.503125 23.4627 \n","L 95.203125 21.76395 \n","L 116.903125 18.4797 \n","L 138.603125 22.3302 \n","L 160.303125 18.00405 \n","L 182.003125 21.1071 \n","L 203.703125 24.1422 \n","L 225.403125 16.19205 \n","\" clip-path=\"url(#p85b7155709)\" style=\"fill: none; stroke-dasharray: 9.6,2.4,1.5,2.4; stroke-dashoffset: 0; stroke: #008000; stroke-width: 1.5\"/>\n","   </g>\n","   <g id=\"patch_3\">\n","    <path d=\"M 30.103125 143.1 \n","L 30.103125 7.2 \n","\" style=\"fill: none; stroke: #000000; stroke-width: 0.8; stroke-linejoin: miter; stroke-linecap: square\"/>\n","   </g>\n","   <g id=\"patch_4\">\n","    <path d=\"M 225.403125 143.1 \n","L 225.403125 7.2 \n","\" style=\"fill: none; stroke: #000000; stroke-width: 0.8; stroke-linejoin: miter; stroke-linecap: square\"/>\n","   </g>\n","   <g id=\"patch_5\">\n","    <path d=\"M 30.103125 143.1 \n","L 225.403125 143.1 \n","\" style=\"fill: none; stroke: #000000; stroke-width: 0.8; stroke-linejoin: miter; stroke-linecap: square\"/>\n","   </g>\n","   <g id=\"patch_6\">\n","    <path d=\"M 30.103125 7.2 \n","L 225.403125 7.2 \n","\" style=\"fill: none; stroke: #000000; stroke-width: 0.8; stroke-linejoin: miter; stroke-linecap: square\"/>\n","   </g>\n","   <g id=\"legend_1\">\n","    <g id=\"patch_7\">\n","     <path d=\"M 140.634375 98.667187 \n","L 218.403125 98.667187 \n","Q 220.403125 98.667187 220.403125 96.667187 \n","L 220.403125 53.632812 \n","Q 220.403125 51.632812 218.403125 51.632812 \n","L 140.634375 51.632812 \n","Q 138.634375 51.632812 138.634375 53.632812 \n","L 138.634375 96.667187 \n","Q 138.634375 98.667187 140.634375 98.667187 \n","z\n","\" style=\"fill: #ffffff; opacity: 0.8; stroke: #cccccc; stroke-linejoin: miter\"/>\n","    </g>\n","    <g id=\"line2d_20\">\n","     <path d=\"M 142.634375 59.73125 \n","L 152.634375 59.73125 \n","L 162.634375 59.73125 \n","\" style=\"fill: none; stroke: #1f77b4; stroke-width: 1.5; stroke-linecap: square\"/>\n","    </g>\n","    <g id=\"text_10\">\n","     <!-- train loss -->\n","     <g transform=\"translate(170.634375 63.23125)scale(0.1 -0.1)\">\n","      <defs>\n","       <path id=\"DejaVuSans-74\" d=\"M 1172 4494 \n","L 1172 3500 \n","L 2356 3500 \n","L 2356 3053 \n","L 1172 3053 \n","L 1172 1153 \n","Q 1172 725 1289 603 \n","Q 1406 481 1766 481 \n","L 2356 481 \n","L 2356 0 \n","L 1766 0 \n","Q 1100 0 847 248 \n","Q 594 497 594 1153 \n","L 594 3053 \n","L 172 3053 \n","L 172 3500 \n","L 594 3500 \n","L 594 4494 \n","L 1172 4494 \n","z\n","\" transform=\"scale(0.015625)\"/>\n","       <path id=\"DejaVuSans-72\" d=\"M 2631 2963 \n","Q 2534 3019 2420 3045 \n","Q 2306 3072 2169 3072 \n","Q 1681 3072 1420 2755 \n","Q 1159 2438 1159 1844 \n","L 1159 0 \n","L 581 0 \n","L 581 3500 \n","L 1159 3500 \n","L 1159 2956 \n","Q 1341 3275 1631 3429 \n","Q 1922 3584 2338 3584 \n","Q 2397 3584 2469 3576 \n","Q 2541 3569 2628 3553 \n","L 2631 2963 \n","z\n","\" transform=\"scale(0.015625)\"/>\n","       <path id=\"DejaVuSans-61\" d=\"M 2194 1759 \n","Q 1497 1759 1228 1600 \n","Q 959 1441 959 1056 \n","Q 959 750 1161 570 \n","Q 1363 391 1709 391 \n","Q 2188 391 2477 730 \n","Q 2766 1069 2766 1631 \n","L 2766 1759 \n","L 2194 1759 \n","z\n","M 3341 1997 \n","L 3341 0 \n","L 2766 0 \n","L 2766 531 \n","Q 2569 213 2275 61 \n","Q 1981 -91 1556 -91 \n","Q 1019 -91 701 211 \n","Q 384 513 384 1019 \n","Q 384 1609 779 1909 \n","Q 1175 2209 1959 2209 \n","L 2766 2209 \n","L 2766 2266 \n","Q 2766 2663 2505 2880 \n","Q 2244 3097 1772 3097 \n","Q 1472 3097 1187 3025 \n","Q 903 2953 641 2809 \n","L 641 3341 \n","Q 956 3463 1253 3523 \n","Q 1550 3584 1831 3584 \n","Q 2591 3584 2966 3190 \n","Q 3341 2797 3341 1997 \n","z\n","\" transform=\"scale(0.015625)\"/>\n","       <path id=\"DejaVuSans-69\" d=\"M 603 3500 \n","L 1178 3500 \n","L 1178 0 \n","L 603 0 \n","L 603 3500 \n","z\n","M 603 4863 \n","L 1178 4863 \n","L 1178 4134 \n","L 603 4134 \n","L 603 4863 \n","z\n","\" transform=\"scale(0.015625)\"/>\n","       <path id=\"DejaVuSans-6e\" d=\"M 3513 2113 \n","L 3513 0 \n","L 2938 0 \n","L 2938 2094 \n","Q 2938 2591 2744 2837 \n","Q 2550 3084 2163 3084 \n","Q 1697 3084 1428 2787 \n","Q 1159 2491 1159 1978 \n","L 1159 0 \n","L 581 0 \n","L 581 3500 \n","L 1159 3500 \n","L 1159 2956 \n","Q 1366 3272 1645 3428 \n","Q 1925 3584 2291 3584 \n","Q 2894 3584 3203 3211 \n","Q 3513 2838 3513 2113 \n","z\n","\" transform=\"scale(0.015625)\"/>\n","       <path id=\"DejaVuSans-20\" transform=\"scale(0.015625)\"/>\n","       <path id=\"DejaVuSans-6c\" d=\"M 603 4863 \n","L 1178 4863 \n","L 1178 0 \n","L 603 0 \n","L 603 4863 \n","z\n","\" transform=\"scale(0.015625)\"/>\n","       <path id=\"DejaVuSans-73\" d=\"M 2834 3397 \n","L 2834 2853 \n","Q 2591 2978 2328 3040 \n","Q 2066 3103 1784 3103 \n","Q 1356 3103 1142 2972 \n","Q 928 2841 928 2578 \n","Q 928 2378 1081 2264 \n","Q 1234 2150 1697 2047 \n","L 1894 2003 \n","Q 2506 1872 2764 1633 \n","Q 3022 1394 3022 966 \n","Q 3022 478 2636 193 \n","Q 2250 -91 1575 -91 \n","Q 1294 -91 989 -36 \n","Q 684 19 347 128 \n","L 347 722 \n","Q 666 556 975 473 \n","Q 1284 391 1588 391 \n","Q 1994 391 2212 530 \n","Q 2431 669 2431 922 \n","Q 2431 1156 2273 1281 \n","Q 2116 1406 1581 1522 \n","L 1381 1569 \n","Q 847 1681 609 1914 \n","Q 372 2147 372 2553 \n","Q 372 3047 722 3315 \n","Q 1072 3584 1716 3584 \n","Q 2034 3584 2315 3537 \n","Q 2597 3491 2834 3397 \n","z\n","\" transform=\"scale(0.015625)\"/>\n","      </defs>\n","      <use xlink:href=\"#DejaVuSans-74\"/>\n","      <use xlink:href=\"#DejaVuSans-72\" x=\"39.208984\"/>\n","      <use xlink:href=\"#DejaVuSans-61\" x=\"80.322266\"/>\n","      <use xlink:href=\"#DejaVuSans-69\" x=\"141.601562\"/>\n","      <use xlink:href=\"#DejaVuSans-6e\" x=\"169.384766\"/>\n","      <use xlink:href=\"#DejaVuSans-20\" x=\"232.763672\"/>\n","      <use xlink:href=\"#DejaVuSans-6c\" x=\"264.550781\"/>\n","      <use xlink:href=\"#DejaVuSans-6f\" x=\"292.333984\"/>\n","      <use xlink:href=\"#DejaVuSans-73\" x=\"353.515625\"/>\n","      <use xlink:href=\"#DejaVuSans-73\" x=\"405.615234\"/>\n","     </g>\n","    </g>\n","    <g id=\"line2d_21\">\n","     <path d=\"M 142.634375 74.409375 \n","L 152.634375 74.409375 \n","L 162.634375 74.409375 \n","\" style=\"fill: none; stroke-dasharray: 5.55,2.4; stroke-dashoffset: 0; stroke: #bf00bf; stroke-width: 1.5\"/>\n","    </g>\n","    <g id=\"text_11\">\n","     <!-- train acc -->\n","     <g transform=\"translate(170.634375 77.909375)scale(0.1 -0.1)\">\n","      <use xlink:href=\"#DejaVuSans-74\"/>\n","      <use xlink:href=\"#DejaVuSans-72\" x=\"39.208984\"/>\n","      <use xlink:href=\"#DejaVuSans-61\" x=\"80.322266\"/>\n","      <use xlink:href=\"#DejaVuSans-69\" x=\"141.601562\"/>\n","      <use xlink:href=\"#DejaVuSans-6e\" x=\"169.384766\"/>\n","      <use xlink:href=\"#DejaVuSans-20\" x=\"232.763672\"/>\n","      <use xlink:href=\"#DejaVuSans-61\" x=\"264.550781\"/>\n","      <use xlink:href=\"#DejaVuSans-63\" x=\"325.830078\"/>\n","      <use xlink:href=\"#DejaVuSans-63\" x=\"380.810547\"/>\n","     </g>\n","    </g>\n","    <g id=\"line2d_22\">\n","     <path d=\"M 142.634375 89.0875 \n","L 152.634375 89.0875 \n","L 162.634375 89.0875 \n","\" style=\"fill: none; stroke-dasharray: 9.6,2.4,1.5,2.4; stroke-dashoffset: 0; stroke: #008000; stroke-width: 1.5\"/>\n","    </g>\n","    <g id=\"text_12\">\n","     <!-- test acc -->\n","     <g transform=\"translate(170.634375 92.5875)scale(0.1 -0.1)\">\n","      <use xlink:href=\"#DejaVuSans-74\"/>\n","      <use xlink:href=\"#DejaVuSans-65\" x=\"39.208984\"/>\n","      <use xlink:href=\"#DejaVuSans-73\" x=\"100.732422\"/>\n","      <use xlink:href=\"#DejaVuSans-74\" x=\"152.832031\"/>\n","      <use xlink:href=\"#DejaVuSans-20\" x=\"192.041016\"/>\n","      <use xlink:href=\"#DejaVuSans-61\" x=\"223.828125\"/>\n","      <use xlink:href=\"#DejaVuSans-63\" x=\"285.107422\"/>\n","      <use xlink:href=\"#DejaVuSans-63\" x=\"340.087891\"/>\n","     </g>\n","    </g>\n","   </g>\n","  </g>\n"," </g>\n"," <defs>\n","  <clipPath id=\"p85b7155709\">\n","   <rect x=\"30.103125\" y=\"7.2\" width=\"195.3\" height=\"135.9\"/>\n","  </clipPath>\n"," </defs>\n","</svg>\n"],"text/plain":["<Figure size 252x180 with 1 Axes>"]},"metadata":{"needs_background":"light"},"output_type":"display_data"}],"source":["num_epochs, lr, batch_size = 10, 0.5, 256\n","loss = nn.CrossEntropyLoss(reduction='none')\n","train_iter, test_iter = d2l.load_data_fashion_mnist(batch_size)\n","trainer = torch.optim.SGD(net.parameters(), lr=lr)\n","d2l.train_ch3(net, train_iter, test_iter, loss, num_epochs, trainer)"],"id":"74f2a8aa"},{"cell_type":"markdown","metadata":{"origin_pos":20,"id":"44fb199b"},"source":["## [**简洁实现**]\n","\n","对于深度学习框架的高级API，我们只需在每个全连接层之后添加一个`Dropout`层，\n","将暂退概率作为唯一的参数传递给它的构造函数。\n","在训练时，`Dropout`层将根据指定的暂退概率随机丢弃上一层的输出（相当于下一层的输入）。\n","在测试时，`Dropout`层仅传递数据。\n"],"id":"44fb199b"},{"cell_type":"code","execution_count":null,"metadata":{"execution":{"iopub.execute_input":"2022-07-31T02:19:44.140715Z","iopub.status.busy":"2022-07-31T02:19:44.139974Z","iopub.status.idle":"2022-07-31T02:19:44.151183Z","shell.execute_reply":"2022-07-31T02:19:44.150536Z"},"origin_pos":22,"tab":["pytorch"],"id":"172c634a"},"outputs":[],"source":["net = nn.Sequential(nn.Flatten(),\n","        nn.Linear(784, 256),\n","        nn.ReLU(),\n","        # 在第一个全连接层之后添加一个dropout层\n","        nn.Dropout(dropout1),\n","        nn.Linear(256, 256),\n","        nn.ReLU(),\n","        # 在第二个全连接层之后添加一个dropout层\n","        nn.Dropout(dropout2),\n","        nn.Linear(256, 10))\n","\n","def init_weights(m):\n","    if type(m) == nn.Linear:\n","        nn.init.normal_(m.weight, std=0.01)\n","\n","net.apply(init_weights);"],"id":"172c634a"},{"cell_type":"markdown","metadata":{"origin_pos":24,"id":"01e7c851"},"source":["接下来，我们[**对模型进行训练和测试**]。\n"],"id":"01e7c851"},{"cell_type":"code","execution_count":null,"metadata":{"execution":{"iopub.execute_input":"2022-07-31T02:19:44.154337Z","iopub.status.busy":"2022-07-31T02:19:44.153925Z","iopub.status.idle":"2022-07-31T02:20:24.010076Z","shell.execute_reply":"2022-07-31T02:20:24.009289Z"},"origin_pos":26,"tab":["pytorch"],"id":"fa9409fe","outputId":"3f2a02ec-44a5-4a22-a4bf-c2648e525914"},"outputs":[{"data":{"image/svg+xml":["<?xml version=\"1.0\" encoding=\"utf-8\" standalone=\"no\"?>\n","<!DOCTYPE svg PUBLIC \"-//W3C//DTD SVG 1.1//EN\"\n","  \"http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd\">\n","<svg xmlns:xlink=\"http://www.w3.org/1999/xlink\" width=\"238.965625pt\" height=\"180.65625pt\" viewBox=\"0 0 238.965625 180.65625\" xmlns=\"http://www.w3.org/2000/svg\" version=\"1.1\">\n"," <metadata>\n","  <rdf:RDF xmlns:dc=\"http://purl.org/dc/elements/1.1/\" xmlns:cc=\"http://creativecommons.org/ns#\" xmlns:rdf=\"http://www.w3.org/1999/02/22-rdf-syntax-ns#\">\n","   <cc:Work>\n","    <dc:type rdf:resource=\"http://purl.org/dc/dcmitype/StillImage\"/>\n","    <dc:date>2022-07-31T02:20:23.967874</dc:date>\n","    <dc:format>image/svg+xml</dc:format>\n","    <dc:creator>\n","     <cc:Agent>\n","      <dc:title>Matplotlib v3.5.1, https://matplotlib.org/</dc:title>\n","     </cc:Agent>\n","    </dc:creator>\n","   </cc:Work>\n","  </rdf:RDF>\n"," </metadata>\n"," <defs>\n","  <style type=\"text/css\">*{stroke-linejoin: round; stroke-linecap: butt}</style>\n"," </defs>\n"," <g id=\"figure_1\">\n","  <g id=\"patch_1\">\n","   <path d=\"M 0 180.65625 \n","L 238.965625 180.65625 \n","L 238.965625 0 \n","L 0 0 \n","L 0 180.65625 \n","z\n","\" style=\"fill: none\"/>\n","  </g>\n","  <g id=\"axes_1\">\n","   <g id=\"patch_2\">\n","    <path d=\"M 30.103125 143.1 \n","L 225.403125 143.1 \n","L 225.403125 7.2 \n","L 30.103125 7.2 \n","z\n","\" style=\"fill: #ffffff\"/>\n","   </g>\n","   <g id=\"matplotlib.axis_1\">\n","    <g id=\"xtick_1\">\n","     <g id=\"line2d_1\">\n","      <path d=\"M 51.803125 143.1 \n","L 51.803125 7.2 \n","\" clip-path=\"url(#pd1bee28a2c)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n","     </g>\n","     <g id=\"line2d_2\">\n","      <defs>\n","       <path id=\"maf0193573e\" d=\"M 0 0 \n","L 0 3.5 \n","\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n","      </defs>\n","      <g>\n","       <use xlink:href=\"#maf0193573e\" x=\"51.803125\" y=\"143.1\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n","      </g>\n","     </g>\n","     <g id=\"text_1\">\n","      <!-- 2 -->\n","      <g transform=\"translate(48.621875 157.698438)scale(0.1 -0.1)\">\n","       <defs>\n","        <path id=\"DejaVuSans-32\" d=\"M 1228 531 \n","L 3431 531 \n","L 3431 0 \n","L 469 0 \n","L 469 531 \n","Q 828 903 1448 1529 \n","Q 2069 2156 2228 2338 \n","Q 2531 2678 2651 2914 \n","Q 2772 3150 2772 3378 \n","Q 2772 3750 2511 3984 \n","Q 2250 4219 1831 4219 \n","Q 1534 4219 1204 4116 \n","Q 875 4013 500 3803 \n","L 500 4441 \n","Q 881 4594 1212 4672 \n","Q 1544 4750 1819 4750 \n","Q 2544 4750 2975 4387 \n","Q 3406 4025 3406 3419 \n","Q 3406 3131 3298 2873 \n","Q 3191 2616 2906 2266 \n","Q 2828 2175 2409 1742 \n","Q 1991 1309 1228 531 \n","z\n","\" transform=\"scale(0.015625)\"/>\n","       </defs>\n","       <use xlink:href=\"#DejaVuSans-32\"/>\n","      </g>\n","     </g>\n","    </g>\n","    <g id=\"xtick_2\">\n","     <g id=\"line2d_3\">\n","      <path d=\"M 95.203125 143.1 \n","L 95.203125 7.2 \n","\" clip-path=\"url(#pd1bee28a2c)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n","     </g>\n","     <g id=\"line2d_4\">\n","      <g>\n","       <use xlink:href=\"#maf0193573e\" x=\"95.203125\" y=\"143.1\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n","      </g>\n","     </g>\n","     <g id=\"text_2\">\n","      <!-- 4 -->\n","      <g transform=\"translate(92.021875 157.698438)scale(0.1 -0.1)\">\n","       <defs>\n","        <path id=\"DejaVuSans-34\" d=\"M 2419 4116 \n","L 825 1625 \n","L 2419 1625 \n","L 2419 4116 \n","z\n","M 2253 4666 \n","L 3047 4666 \n","L 3047 1625 \n","L 3713 1625 \n","L 3713 1100 \n","L 3047 1100 \n","L 3047 0 \n","L 2419 0 \n","L 2419 1100 \n","L 313 1100 \n","L 313 1709 \n","L 2253 4666 \n","z\n","\" transform=\"scale(0.015625)\"/>\n","       </defs>\n","       <use xlink:href=\"#DejaVuSans-34\"/>\n","      </g>\n","     </g>\n","    </g>\n","    <g id=\"xtick_3\">\n","     <g id=\"line2d_5\">\n","      <path d=\"M 138.603125 143.1 \n","L 138.603125 7.2 \n","\" clip-path=\"url(#pd1bee28a2c)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n","     </g>\n","     <g id=\"line2d_6\">\n","      <g>\n","       <use xlink:href=\"#maf0193573e\" x=\"138.603125\" y=\"143.1\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n","      </g>\n","     </g>\n","     <g id=\"text_3\">\n","      <!-- 6 -->\n","      <g transform=\"translate(135.421875 157.698438)scale(0.1 -0.1)\">\n","       <defs>\n","        <path id=\"DejaVuSans-36\" d=\"M 2113 2584 \n","Q 1688 2584 1439 2293 \n","Q 1191 2003 1191 1497 \n","Q 1191 994 1439 701 \n","Q 1688 409 2113 409 \n","Q 2538 409 2786 701 \n","Q 3034 994 3034 1497 \n","Q 3034 2003 2786 2293 \n","Q 2538 2584 2113 2584 \n","z\n","M 3366 4563 \n","L 3366 3988 \n","Q 3128 4100 2886 4159 \n","Q 2644 4219 2406 4219 \n","Q 1781 4219 1451 3797 \n","Q 1122 3375 1075 2522 \n","Q 1259 2794 1537 2939 \n","Q 1816 3084 2150 3084 \n","Q 2853 3084 3261 2657 \n","Q 3669 2231 3669 1497 \n","Q 3669 778 3244 343 \n","Q 2819 -91 2113 -91 \n","Q 1303 -91 875 529 \n","Q 447 1150 447 2328 \n","Q 447 3434 972 4092 \n","Q 1497 4750 2381 4750 \n","Q 2619 4750 2861 4703 \n","Q 3103 4656 3366 4563 \n","z\n","\" transform=\"scale(0.015625)\"/>\n","       </defs>\n","       <use xlink:href=\"#DejaVuSans-36\"/>\n","      </g>\n","     </g>\n","    </g>\n","    <g id=\"xtick_4\">\n","     <g id=\"line2d_7\">\n","      <path d=\"M 182.003125 143.1 \n","L 182.003125 7.2 \n","\" clip-path=\"url(#pd1bee28a2c)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n","     </g>\n","     <g id=\"line2d_8\">\n","      <g>\n","       <use xlink:href=\"#maf0193573e\" x=\"182.003125\" y=\"143.1\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n","      </g>\n","     </g>\n","     <g id=\"text_4\">\n","      <!-- 8 -->\n","      <g transform=\"translate(178.821875 157.698438)scale(0.1 -0.1)\">\n","       <defs>\n","        <path id=\"DejaVuSans-38\" d=\"M 2034 2216 \n","Q 1584 2216 1326 1975 \n","Q 1069 1734 1069 1313 \n","Q 1069 891 1326 650 \n","Q 1584 409 2034 409 \n","Q 2484 409 2743 651 \n","Q 3003 894 3003 1313 \n","Q 3003 1734 2745 1975 \n","Q 2488 2216 2034 2216 \n","z\n","M 1403 2484 \n","Q 997 2584 770 2862 \n","Q 544 3141 544 3541 \n","Q 544 4100 942 4425 \n","Q 1341 4750 2034 4750 \n","Q 2731 4750 3128 4425 \n","Q 3525 4100 3525 3541 \n","Q 3525 3141 3298 2862 \n","Q 3072 2584 2669 2484 \n","Q 3125 2378 3379 2068 \n","Q 3634 1759 3634 1313 \n","Q 3634 634 3220 271 \n","Q 2806 -91 2034 -91 \n","Q 1263 -91 848 271 \n","Q 434 634 434 1313 \n","Q 434 1759 690 2068 \n","Q 947 2378 1403 2484 \n","z\n","M 1172 3481 \n","Q 1172 3119 1398 2916 \n","Q 1625 2713 2034 2713 \n","Q 2441 2713 2670 2916 \n","Q 2900 3119 2900 3481 \n","Q 2900 3844 2670 4047 \n","Q 2441 4250 2034 4250 \n","Q 1625 4250 1398 4047 \n","Q 1172 3844 1172 3481 \n","z\n","\" transform=\"scale(0.015625)\"/>\n","       </defs>\n","       <use xlink:href=\"#DejaVuSans-38\"/>\n","      </g>\n","     </g>\n","    </g>\n","    <g id=\"xtick_5\">\n","     <g id=\"line2d_9\">\n","      <path d=\"M 225.403125 143.1 \n","L 225.403125 7.2 \n","\" clip-path=\"url(#pd1bee28a2c)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n","     </g>\n","     <g id=\"line2d_10\">\n","      <g>\n","       <use xlink:href=\"#maf0193573e\" x=\"225.403125\" y=\"143.1\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n","      </g>\n","     </g>\n","     <g id=\"text_5\">\n","      <!-- 10 -->\n","      <g transform=\"translate(219.040625 157.698438)scale(0.1 -0.1)\">\n","       <defs>\n","        <path id=\"DejaVuSans-31\" d=\"M 794 531 \n","L 1825 531 \n","L 1825 4091 \n","L 703 3866 \n","L 703 4441 \n","L 1819 4666 \n","L 2450 4666 \n","L 2450 531 \n","L 3481 531 \n","L 3481 0 \n","L 794 0 \n","L 794 531 \n","z\n","\" transform=\"scale(0.015625)\"/>\n","        <path id=\"DejaVuSans-30\" d=\"M 2034 4250 \n","Q 1547 4250 1301 3770 \n","Q 1056 3291 1056 2328 \n","Q 1056 1369 1301 889 \n","Q 1547 409 2034 409 \n","Q 2525 409 2770 889 \n","Q 3016 1369 3016 2328 \n","Q 3016 3291 2770 3770 \n","Q 2525 4250 2034 4250 \n","z\n","M 2034 4750 \n","Q 2819 4750 3233 4129 \n","Q 3647 3509 3647 2328 \n","Q 3647 1150 3233 529 \n","Q 2819 -91 2034 -91 \n","Q 1250 -91 836 529 \n","Q 422 1150 422 2328 \n","Q 422 3509 836 4129 \n","Q 1250 4750 2034 4750 \n","z\n","\" transform=\"scale(0.015625)\"/>\n","       </defs>\n","       <use xlink:href=\"#DejaVuSans-31\"/>\n","       <use xlink:href=\"#DejaVuSans-30\" x=\"63.623047\"/>\n","      </g>\n","     </g>\n","    </g>\n","    <g id=\"text_6\">\n","     <!-- epoch -->\n","     <g transform=\"translate(112.525 171.376563)scale(0.1 -0.1)\">\n","      <defs>\n","       <path id=\"DejaVuSans-65\" d=\"M 3597 1894 \n","L 3597 1613 \n","L 953 1613 \n","Q 991 1019 1311 708 \n","Q 1631 397 2203 397 \n","Q 2534 397 2845 478 \n","Q 3156 559 3463 722 \n","L 3463 178 \n","Q 3153 47 2828 -22 \n","Q 2503 -91 2169 -91 \n","Q 1331 -91 842 396 \n","Q 353 884 353 1716 \n","Q 353 2575 817 3079 \n","Q 1281 3584 2069 3584 \n","Q 2775 3584 3186 3129 \n","Q 3597 2675 3597 1894 \n","z\n","M 3022 2063 \n","Q 3016 2534 2758 2815 \n","Q 2500 3097 2075 3097 \n","Q 1594 3097 1305 2825 \n","Q 1016 2553 972 2059 \n","L 3022 2063 \n","z\n","\" transform=\"scale(0.015625)\"/>\n","       <path id=\"DejaVuSans-70\" d=\"M 1159 525 \n","L 1159 -1331 \n","L 581 -1331 \n","L 581 3500 \n","L 1159 3500 \n","L 1159 2969 \n","Q 1341 3281 1617 3432 \n","Q 1894 3584 2278 3584 \n","Q 2916 3584 3314 3078 \n","Q 3713 2572 3713 1747 \n","Q 3713 922 3314 415 \n","Q 2916 -91 2278 -91 \n","Q 1894 -91 1617 61 \n","Q 1341 213 1159 525 \n","z\n","M 3116 1747 \n","Q 3116 2381 2855 2742 \n","Q 2594 3103 2138 3103 \n","Q 1681 3103 1420 2742 \n","Q 1159 2381 1159 1747 \n","Q 1159 1113 1420 752 \n","Q 1681 391 2138 391 \n","Q 2594 391 2855 752 \n","Q 3116 1113 3116 1747 \n","z\n","\" transform=\"scale(0.015625)\"/>\n","       <path id=\"DejaVuSans-6f\" d=\"M 1959 3097 \n","Q 1497 3097 1228 2736 \n","Q 959 2375 959 1747 \n","Q 959 1119 1226 758 \n","Q 1494 397 1959 397 \n","Q 2419 397 2687 759 \n","Q 2956 1122 2956 1747 \n","Q 2956 2369 2687 2733 \n","Q 2419 3097 1959 3097 \n","z\n","M 1959 3584 \n","Q 2709 3584 3137 3096 \n","Q 3566 2609 3566 1747 \n","Q 3566 888 3137 398 \n","Q 2709 -91 1959 -91 \n","Q 1206 -91 779 398 \n","Q 353 888 353 1747 \n","Q 353 2609 779 3096 \n","Q 1206 3584 1959 3584 \n","z\n","\" transform=\"scale(0.015625)\"/>\n","       <path id=\"DejaVuSans-63\" d=\"M 3122 3366 \n","L 3122 2828 \n","Q 2878 2963 2633 3030 \n","Q 2388 3097 2138 3097 \n","Q 1578 3097 1268 2742 \n","Q 959 2388 959 1747 \n","Q 959 1106 1268 751 \n","Q 1578 397 2138 397 \n","Q 2388 397 2633 464 \n","Q 2878 531 3122 666 \n","L 3122 134 \n","Q 2881 22 2623 -34 \n","Q 2366 -91 2075 -91 \n","Q 1284 -91 818 406 \n","Q 353 903 353 1747 \n","Q 353 2603 823 3093 \n","Q 1294 3584 2113 3584 \n","Q 2378 3584 2631 3529 \n","Q 2884 3475 3122 3366 \n","z\n","\" transform=\"scale(0.015625)\"/>\n","       <path id=\"DejaVuSans-68\" d=\"M 3513 2113 \n","L 3513 0 \n","L 2938 0 \n","L 2938 2094 \n","Q 2938 2591 2744 2837 \n","Q 2550 3084 2163 3084 \n","Q 1697 3084 1428 2787 \n","Q 1159 2491 1159 1978 \n","L 1159 0 \n","L 581 0 \n","L 581 4863 \n","L 1159 4863 \n","L 1159 2956 \n","Q 1366 3272 1645 3428 \n","Q 1925 3584 2291 3584 \n","Q 2894 3584 3203 3211 \n","Q 3513 2838 3513 2113 \n","z\n","\" transform=\"scale(0.015625)\"/>\n","      </defs>\n","      <use xlink:href=\"#DejaVuSans-65\"/>\n","      <use xlink:href=\"#DejaVuSans-70\" x=\"61.523438\"/>\n","      <use xlink:href=\"#DejaVuSans-6f\" x=\"125\"/>\n","      <use xlink:href=\"#DejaVuSans-63\" x=\"186.181641\"/>\n","      <use xlink:href=\"#DejaVuSans-68\" x=\"241.162109\"/>\n","     </g>\n","    </g>\n","   </g>\n","   <g id=\"matplotlib.axis_2\">\n","    <g id=\"ytick_1\">\n","     <g id=\"line2d_11\">\n","      <path d=\"M 30.103125 120.45 \n","L 225.403125 120.45 \n","\" clip-path=\"url(#pd1bee28a2c)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n","     </g>\n","     <g id=\"line2d_12\">\n","      <defs>\n","       <path id=\"mad57de4b3a\" d=\"M 0 0 \n","L -3.5 0 \n","\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n","      </defs>\n","      <g>\n","       <use xlink:href=\"#mad57de4b3a\" x=\"30.103125\" y=\"120.45\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n","      </g>\n","     </g>\n","     <g id=\"text_7\">\n","      <!-- 0.4 -->\n","      <g transform=\"translate(7.2 124.249219)scale(0.1 -0.1)\">\n","       <defs>\n","        <path id=\"DejaVuSans-2e\" d=\"M 684 794 \n","L 1344 794 \n","L 1344 0 \n","L 684 0 \n","L 684 794 \n","z\n","\" transform=\"scale(0.015625)\"/>\n","       </defs>\n","       <use xlink:href=\"#DejaVuSans-30\"/>\n","       <use xlink:href=\"#DejaVuSans-2e\" x=\"63.623047\"/>\n","       <use xlink:href=\"#DejaVuSans-34\" x=\"95.410156\"/>\n","      </g>\n","     </g>\n","    </g>\n","    <g id=\"ytick_2\">\n","     <g id=\"line2d_13\">\n","      <path d=\"M 30.103125 75.15 \n","L 225.403125 75.15 \n","\" clip-path=\"url(#pd1bee28a2c)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n","     </g>\n","     <g id=\"line2d_14\">\n","      <g>\n","       <use xlink:href=\"#mad57de4b3a\" x=\"30.103125\" y=\"75.15\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n","      </g>\n","     </g>\n","     <g id=\"text_8\">\n","      <!-- 0.6 -->\n","      <g transform=\"translate(7.2 78.949219)scale(0.1 -0.1)\">\n","       <use xlink:href=\"#DejaVuSans-30\"/>\n","       <use xlink:href=\"#DejaVuSans-2e\" x=\"63.623047\"/>\n","       <use xlink:href=\"#DejaVuSans-36\" x=\"95.410156\"/>\n","      </g>\n","     </g>\n","    </g>\n","    <g id=\"ytick_3\">\n","     <g id=\"line2d_15\">\n","      <path d=\"M 30.103125 29.85 \n","L 225.403125 29.85 \n","\" clip-path=\"url(#pd1bee28a2c)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n","     </g>\n","     <g id=\"line2d_16\">\n","      <g>\n","       <use xlink:href=\"#mad57de4b3a\" x=\"30.103125\" y=\"29.85\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n","      </g>\n","     </g>\n","     <g id=\"text_9\">\n","      <!-- 0.8 -->\n","      <g transform=\"translate(7.2 33.649219)scale(0.1 -0.1)\">\n","       <use xlink:href=\"#DejaVuSans-30\"/>\n","       <use xlink:href=\"#DejaVuSans-2e\" x=\"63.623047\"/>\n","       <use xlink:href=\"#DejaVuSans-38\" x=\"95.410156\"/>\n","      </g>\n","     </g>\n","    </g>\n","   </g>\n","   <g id=\"line2d_17\">\n","    <path d=\"M 37.289553 -1 \n","L 51.803125 81.110891 \n","L 73.503125 101.7247 \n","L 95.203125 111.150033 \n","L 116.903125 117.81057 \n","L 138.603125 122.036506 \n","L 160.303125 126.424096 \n","L 182.003125 128.730303 \n","L 203.703125 131.163028 \n","L 225.403125 134.32983 \n","\" clip-path=\"url(#pd1bee28a2c)\" style=\"fill: none; stroke: #1f77b4; stroke-width: 1.5; stroke-linecap: square\"/>\n","   </g>\n","   <g id=\"line2d_18\">\n","    <path d=\"M 30.103125 82.42065 \n","L 51.803125 33.04365 \n","L 73.503125 24.7764 \n","L 95.203125 21.103325 \n","L 116.903125 18.68355 \n","L 138.603125 17.1207 \n","L 160.303125 15.6258 \n","L 182.003125 14.92365 \n","L 203.703125 13.844 \n","L 225.403125 12.74925 \n","\" clip-path=\"url(#pd1bee28a2c)\" style=\"fill: none; stroke-dasharray: 5.55,2.4; stroke-dashoffset: 0; stroke: #bf00bf; stroke-width: 1.5\"/>\n","   </g>\n","   <g id=\"line2d_19\">\n","    <path d=\"M 30.103125 42.0357 \n","L 51.803125 32.22825 \n","L 73.503125 35.5125 \n","L 95.203125 23.3721 \n","L 116.903125 22.71525 \n","L 138.603125 31.45815 \n","L 160.303125 15.46725 \n","L 182.003125 17.05275 \n","L 203.703125 21.62805 \n","L 225.403125 19.18185 \n","\" clip-path=\"url(#pd1bee28a2c)\" style=\"fill: none; stroke-dasharray: 9.6,2.4,1.5,2.4; stroke-dashoffset: 0; stroke: #008000; stroke-width: 1.5\"/>\n","   </g>\n","   <g id=\"patch_3\">\n","    <path d=\"M 30.103125 143.1 \n","L 30.103125 7.2 \n","\" style=\"fill: none; stroke: #000000; stroke-width: 0.8; stroke-linejoin: miter; stroke-linecap: square\"/>\n","   </g>\n","   <g id=\"patch_4\">\n","    <path d=\"M 225.403125 143.1 \n","L 225.403125 7.2 \n","\" style=\"fill: none; stroke: #000000; stroke-width: 0.8; stroke-linejoin: miter; stroke-linecap: square\"/>\n","   </g>\n","   <g id=\"patch_5\">\n","    <path d=\"M 30.103125 143.1 \n","L 225.403125 143.1 \n","\" style=\"fill: none; stroke: #000000; stroke-width: 0.8; stroke-linejoin: miter; stroke-linecap: square\"/>\n","   </g>\n","   <g id=\"patch_6\">\n","    <path d=\"M 30.103125 7.2 \n","L 225.403125 7.2 \n","\" style=\"fill: none; stroke: #000000; stroke-width: 0.8; stroke-linejoin: miter; stroke-linecap: square\"/>\n","   </g>\n","   <g id=\"legend_1\">\n","    <g id=\"patch_7\">\n","     <path d=\"M 140.634375 98.667187 \n","L 218.403125 98.667187 \n","Q 220.403125 98.667187 220.403125 96.667187 \n","L 220.403125 53.632812 \n","Q 220.403125 51.632812 218.403125 51.632812 \n","L 140.634375 51.632812 \n","Q 138.634375 51.632812 138.634375 53.632812 \n","L 138.634375 96.667187 \n","Q 138.634375 98.667187 140.634375 98.667187 \n","z\n","\" style=\"fill: #ffffff; opacity: 0.8; stroke: #cccccc; stroke-linejoin: miter\"/>\n","    </g>\n","    <g id=\"line2d_20\">\n","     <path d=\"M 142.634375 59.73125 \n","L 152.634375 59.73125 \n","L 162.634375 59.73125 \n","\" style=\"fill: none; stroke: #1f77b4; stroke-width: 1.5; stroke-linecap: square\"/>\n","    </g>\n","    <g id=\"text_10\">\n","     <!-- train loss -->\n","     <g transform=\"translate(170.634375 63.23125)scale(0.1 -0.1)\">\n","      <defs>\n","       <path id=\"DejaVuSans-74\" d=\"M 1172 4494 \n","L 1172 3500 \n","L 2356 3500 \n","L 2356 3053 \n","L 1172 3053 \n","L 1172 1153 \n","Q 1172 725 1289 603 \n","Q 1406 481 1766 481 \n","L 2356 481 \n","L 2356 0 \n","L 1766 0 \n","Q 1100 0 847 248 \n","Q 594 497 594 1153 \n","L 594 3053 \n","L 172 3053 \n","L 172 3500 \n","L 594 3500 \n","L 594 4494 \n","L 1172 4494 \n","z\n","\" transform=\"scale(0.015625)\"/>\n","       <path id=\"DejaVuSans-72\" d=\"M 2631 2963 \n","Q 2534 3019 2420 3045 \n","Q 2306 3072 2169 3072 \n","Q 1681 3072 1420 2755 \n","Q 1159 2438 1159 1844 \n","L 1159 0 \n","L 581 0 \n","L 581 3500 \n","L 1159 3500 \n","L 1159 2956 \n","Q 1341 3275 1631 3429 \n","Q 1922 3584 2338 3584 \n","Q 2397 3584 2469 3576 \n","Q 2541 3569 2628 3553 \n","L 2631 2963 \n","z\n","\" transform=\"scale(0.015625)\"/>\n","       <path id=\"DejaVuSans-61\" d=\"M 2194 1759 \n","Q 1497 1759 1228 1600 \n","Q 959 1441 959 1056 \n","Q 959 750 1161 570 \n","Q 1363 391 1709 391 \n","Q 2188 391 2477 730 \n","Q 2766 1069 2766 1631 \n","L 2766 1759 \n","L 2194 1759 \n","z\n","M 3341 1997 \n","L 3341 0 \n","L 2766 0 \n","L 2766 531 \n","Q 2569 213 2275 61 \n","Q 1981 -91 1556 -91 \n","Q 1019 -91 701 211 \n","Q 384 513 384 1019 \n","Q 384 1609 779 1909 \n","Q 1175 2209 1959 2209 \n","L 2766 2209 \n","L 2766 2266 \n","Q 2766 2663 2505 2880 \n","Q 2244 3097 1772 3097 \n","Q 1472 3097 1187 3025 \n","Q 903 2953 641 2809 \n","L 641 3341 \n","Q 956 3463 1253 3523 \n","Q 1550 3584 1831 3584 \n","Q 2591 3584 2966 3190 \n","Q 3341 2797 3341 1997 \n","z\n","\" transform=\"scale(0.015625)\"/>\n","       <path id=\"DejaVuSans-69\" d=\"M 603 3500 \n","L 1178 3500 \n","L 1178 0 \n","L 603 0 \n","L 603 3500 \n","z\n","M 603 4863 \n","L 1178 4863 \n","L 1178 4134 \n","L 603 4134 \n","L 603 4863 \n","z\n","\" transform=\"scale(0.015625)\"/>\n","       <path id=\"DejaVuSans-6e\" d=\"M 3513 2113 \n","L 3513 0 \n","L 2938 0 \n","L 2938 2094 \n","Q 2938 2591 2744 2837 \n","Q 2550 3084 2163 3084 \n","Q 1697 3084 1428 2787 \n","Q 1159 2491 1159 1978 \n","L 1159 0 \n","L 581 0 \n","L 581 3500 \n","L 1159 3500 \n","L 1159 2956 \n","Q 1366 3272 1645 3428 \n","Q 1925 3584 2291 3584 \n","Q 2894 3584 3203 3211 \n","Q 3513 2838 3513 2113 \n","z\n","\" transform=\"scale(0.015625)\"/>\n","       <path id=\"DejaVuSans-20\" transform=\"scale(0.015625)\"/>\n","       <path id=\"DejaVuSans-6c\" d=\"M 603 4863 \n","L 1178 4863 \n","L 1178 0 \n","L 603 0 \n","L 603 4863 \n","z\n","\" transform=\"scale(0.015625)\"/>\n","       <path id=\"DejaVuSans-73\" d=\"M 2834 3397 \n","L 2834 2853 \n","Q 2591 2978 2328 3040 \n","Q 2066 3103 1784 3103 \n","Q 1356 3103 1142 2972 \n","Q 928 2841 928 2578 \n","Q 928 2378 1081 2264 \n","Q 1234 2150 1697 2047 \n","L 1894 2003 \n","Q 2506 1872 2764 1633 \n","Q 3022 1394 3022 966 \n","Q 3022 478 2636 193 \n","Q 2250 -91 1575 -91 \n","Q 1294 -91 989 -36 \n","Q 684 19 347 128 \n","L 347 722 \n","Q 666 556 975 473 \n","Q 1284 391 1588 391 \n","Q 1994 391 2212 530 \n","Q 2431 669 2431 922 \n","Q 2431 1156 2273 1281 \n","Q 2116 1406 1581 1522 \n","L 1381 1569 \n","Q 847 1681 609 1914 \n","Q 372 2147 372 2553 \n","Q 372 3047 722 3315 \n","Q 1072 3584 1716 3584 \n","Q 2034 3584 2315 3537 \n","Q 2597 3491 2834 3397 \n","z\n","\" transform=\"scale(0.015625)\"/>\n","      </defs>\n","      <use xlink:href=\"#DejaVuSans-74\"/>\n","      <use xlink:href=\"#DejaVuSans-72\" x=\"39.208984\"/>\n","      <use xlink:href=\"#DejaVuSans-61\" x=\"80.322266\"/>\n","      <use xlink:href=\"#DejaVuSans-69\" x=\"141.601562\"/>\n","      <use xlink:href=\"#DejaVuSans-6e\" x=\"169.384766\"/>\n","      <use xlink:href=\"#DejaVuSans-20\" x=\"232.763672\"/>\n","      <use xlink:href=\"#DejaVuSans-6c\" x=\"264.550781\"/>\n","      <use xlink:href=\"#DejaVuSans-6f\" x=\"292.333984\"/>\n","      <use xlink:href=\"#DejaVuSans-73\" x=\"353.515625\"/>\n","      <use xlink:href=\"#DejaVuSans-73\" x=\"405.615234\"/>\n","     </g>\n","    </g>\n","    <g id=\"line2d_21\">\n","     <path d=\"M 142.634375 74.409375 \n","L 152.634375 74.409375 \n","L 162.634375 74.409375 \n","\" style=\"fill: none; stroke-dasharray: 5.55,2.4; stroke-dashoffset: 0; stroke: #bf00bf; stroke-width: 1.5\"/>\n","    </g>\n","    <g id=\"text_11\">\n","     <!-- train acc -->\n","     <g transform=\"translate(170.634375 77.909375)scale(0.1 -0.1)\">\n","      <use xlink:href=\"#DejaVuSans-74\"/>\n","      <use xlink:href=\"#DejaVuSans-72\" x=\"39.208984\"/>\n","      <use xlink:href=\"#DejaVuSans-61\" x=\"80.322266\"/>\n","      <use xlink:href=\"#DejaVuSans-69\" x=\"141.601562\"/>\n","      <use xlink:href=\"#DejaVuSans-6e\" x=\"169.384766\"/>\n","      <use xlink:href=\"#DejaVuSans-20\" x=\"232.763672\"/>\n","      <use xlink:href=\"#DejaVuSans-61\" x=\"264.550781\"/>\n","      <use xlink:href=\"#DejaVuSans-63\" x=\"325.830078\"/>\n","      <use xlink:href=\"#DejaVuSans-63\" x=\"380.810547\"/>\n","     </g>\n","    </g>\n","    <g id=\"line2d_22\">\n","     <path d=\"M 142.634375 89.0875 \n","L 152.634375 89.0875 \n","L 162.634375 89.0875 \n","\" style=\"fill: none; stroke-dasharray: 9.6,2.4,1.5,2.4; stroke-dashoffset: 0; stroke: #008000; stroke-width: 1.5\"/>\n","    </g>\n","    <g id=\"text_12\">\n","     <!-- test acc -->\n","     <g transform=\"translate(170.634375 92.5875)scale(0.1 -0.1)\">\n","      <use xlink:href=\"#DejaVuSans-74\"/>\n","      <use xlink:href=\"#DejaVuSans-65\" x=\"39.208984\"/>\n","      <use xlink:href=\"#DejaVuSans-73\" x=\"100.732422\"/>\n","      <use xlink:href=\"#DejaVuSans-74\" x=\"152.832031\"/>\n","      <use xlink:href=\"#DejaVuSans-20\" x=\"192.041016\"/>\n","      <use xlink:href=\"#DejaVuSans-61\" x=\"223.828125\"/>\n","      <use xlink:href=\"#DejaVuSans-63\" x=\"285.107422\"/>\n","      <use xlink:href=\"#DejaVuSans-63\" x=\"340.087891\"/>\n","     </g>\n","    </g>\n","   </g>\n","  </g>\n"," </g>\n"," <defs>\n","  <clipPath id=\"pd1bee28a2c\">\n","   <rect x=\"30.103125\" y=\"7.2\" width=\"195.3\" height=\"135.9\"/>\n","  </clipPath>\n"," </defs>\n","</svg>\n"],"text/plain":["<Figure size 252x180 with 1 Axes>"]},"metadata":{"needs_background":"light"},"output_type":"display_data"}],"source":["trainer = torch.optim.SGD(net.parameters(), lr=lr)\n","d2l.train_ch3(net, train_iter, test_iter, loss, num_epochs, trainer)"],"id":"fa9409fe"},{"cell_type":"markdown","metadata":{"origin_pos":28,"id":"8eda5df9"},"source":["## 小结\n","\n","* 暂退法在前向传播过程中，计算每一内部层的同时丢弃一些神经元。\n","* 暂退法可以避免过拟合，它通常与控制权重向量的维数和大小结合使用的。\n","* 暂退法将活性值$h$替换为具有期望值$h$的随机变量。\n","* 暂退法仅在训练期间使用。\n","\n","## 练习\n","\n","1. 如果更改第一层和第二层的暂退法概率，会发生什么情况？具体地说，如果交换这两个层，会发生什么情况？设计一个实验来回答这些问题，定量描述你的结果，并总结定性的结论。\n","1. 增加训练轮数，并将使用暂退法和不使用暂退法时获得的结果进行比较。\n","1. 当应用或不应用暂退法时，每个隐藏层中激活值的方差是多少？绘制一个曲线图，以显示这两个模型的每个隐藏层中激活值的方差是如何随时间变化的。\n","1. 为什么在测试时通常不使用暂退法？\n","1. 以本节中的模型为例，比较使用暂退法和权重衰减的效果。如果同时使用暂退法和权重衰减，会发生什么情况？结果是累加的吗？收益是否减少（或者说更糟）？它们互相抵消了吗？\n","1. 如果我们将暂退法应用到权重矩阵的各个权重，而不是激活值，会发生什么？\n","1. 发明另一种用于在每一层注入随机噪声的技术，该技术不同于标准的暂退法技术。尝试开发一种在Fashion-MNIST数据集（对于固定架构）上性能优于暂退法的方法。\n"],"id":"8eda5df9"},{"cell_type":"markdown","metadata":{"origin_pos":0,"id":"f9bf38d7"},"source":["# 数值稳定性和模型初始化\n",":label:`sec_numerical_stability`\n","\n","到目前为止，我们实现的每个模型都是根据某个预先指定的分布来初始化模型的参数。\n","你可能认为初始化方案是理所当然的，忽略了如何做出这些选择的细节。\n","你甚至可能会觉得，初始化方案的选择并不是特别重要。\n","相反，初始化方案的选择在神经网络学习中起着举足轻重的作用，\n","它对保持数值稳定性至关重要。\n","此外，这些初始化方案的选择可以与非线性激活函数的选择有趣的结合在一起。\n","我们选择哪个函数以及如何初始化参数可以决定优化算法收敛的速度有多快。\n","糟糕选择可能会导致我们在训练时遇到梯度爆炸或梯度消失。\n","在本节中，我们将更详细地探讨这些主题，并讨论一些有用的启发式方法。\n","你会发现这些启发式方法在你的整个深度学习生涯中都很有用。\n","\n","## 梯度消失和梯度爆炸\n","\n","考虑一个具有$L$层、输入$\\mathbf{x}$和输出$\\mathbf{o}$的深层网络。\n","每一层$l$由变换$f_l$定义，\n","该变换的参数为权重$\\mathbf{W}^{(l)}$，\n","其隐藏变量是$\\mathbf{h}^{(l)}$（令 $\\mathbf{h}^{(0)} = \\mathbf{x}$）。\n","我们的网络可以表示为：\n","\n","$$\\mathbf{h}^{(l)} = f_l (\\mathbf{h}^{(l-1)}) \\text{ 因此 } \\mathbf{o} = f_L \\circ \\ldots \\circ f_1(\\mathbf{x}).$$\n","\n","如果所有隐藏变量和输入都是向量，\n","我们可以将$\\mathbf{o}$关于任何一组参数$\\mathbf{W}^{(l)}$的梯度写为下式：\n","\n","$$\\partial_{\\mathbf{W}^{(l)}} \\mathbf{o} = \\underbrace{\\partial_{\\mathbf{h}^{(L-1)}} \\mathbf{h}^{(L)}}_{ \\mathbf{M}^{(L)} \\stackrel{\\mathrm{def}}{=}} \\cdot \\ldots \\cdot \\underbrace{\\partial_{\\mathbf{h}^{(l)}} \\mathbf{h}^{(l+1)}}_{ \\mathbf{M}^{(l+1)} \\stackrel{\\mathrm{def}}{=}} \\underbrace{\\partial_{\\mathbf{W}^{(l)}} \\mathbf{h}^{(l)}}_{ \\mathbf{v}^{(l)} \\stackrel{\\mathrm{def}}{=}}.$$\n","\n","换言之，该梯度是$L-l$个矩阵\n","$\\mathbf{M}^{(L)} \\cdot \\ldots \\cdot \\mathbf{M}^{(l+1)}$\n","与梯度向量 $\\mathbf{v}^{(l)}$的乘积。\n","因此，我们容易受到数值下溢问题的影响.\n","当将太多的概率乘在一起时，这些问题经常会出现。\n","在处理概率时，一个常见的技巧是切换到对数空间，\n","即将数值表示的压力从尾数转移到指数。\n","不幸的是，上面的问题更为严重：\n","最初，矩阵 $\\mathbf{M}^{(l)}$ 可能具有各种各样的特征值。\n","他们可能很小，也可能很大；\n","他们的乘积可能非常大，也可能非常小。\n","\n","不稳定梯度带来的风险不止在于数值表示；\n","不稳定梯度也威胁到我们优化算法的稳定性。\n","我们可能面临一些问题。\n","要么是*梯度爆炸*（gradient exploding）问题：\n","参数更新过大，破坏了模型的稳定收敛；\n","要么是*梯度消失*（gradient vanishing）问题：\n","参数更新过小，在每次更新时几乎不会移动，导致模型无法学习。\n","\n","### (**梯度消失**)\n","\n","曾经sigmoid函数$1/(1 + \\exp(-x))$（ :numref:`sec_mlp`提到过）很流行，\n","因为它类似于阈值函数。\n","由于早期的人工神经网络受到生物神经网络的启发，\n","神经元要么完全激活要么完全不激活（就像生物神经元）的想法很有吸引力。\n","然而，它却是导致梯度消失问题的一个常见的原因，\n","让我们仔细看看sigmoid函数为什么会导致梯度消失。\n"],"id":"f9bf38d7"},{"cell_type":"code","execution_count":null,"metadata":{"execution":{"iopub.execute_input":"2022-07-31T02:43:26.948097Z","iopub.status.busy":"2022-07-31T02:43:26.947591Z","iopub.status.idle":"2022-07-31T02:43:29.091516Z","shell.execute_reply":"2022-07-31T02:43:29.090738Z"},"origin_pos":2,"tab":["pytorch"],"id":"1555bcbf","outputId":"395d9751-7a3a-46cd-fd03-1260dbb2062a"},"outputs":[{"data":{"image/svg+xml":["<?xml version=\"1.0\" encoding=\"utf-8\" standalone=\"no\"?>\n","<!DOCTYPE svg PUBLIC \"-//W3C//DTD SVG 1.1//EN\"\n","  \"http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd\">\n","<svg xmlns:xlink=\"http://www.w3.org/1999/xlink\" width=\"288.403125pt\" height=\"166.978125pt\" viewBox=\"0 0 288.403125 166.978125\" xmlns=\"http://www.w3.org/2000/svg\" version=\"1.1\">\n"," <metadata>\n","  <rdf:RDF xmlns:dc=\"http://purl.org/dc/elements/1.1/\" xmlns:cc=\"http://creativecommons.org/ns#\" xmlns:rdf=\"http://www.w3.org/1999/02/22-rdf-syntax-ns#\">\n","   <cc:Work>\n","    <dc:type rdf:resource=\"http://purl.org/dc/dcmitype/StillImage\"/>\n","    <dc:date>2022-07-31T02:43:29.042225</dc:date>\n","    <dc:format>image/svg+xml</dc:format>\n","    <dc:creator>\n","     <cc:Agent>\n","      <dc:title>Matplotlib v3.5.1, https://matplotlib.org/</dc:title>\n","     </cc:Agent>\n","    </dc:creator>\n","   </cc:Work>\n","  </rdf:RDF>\n"," </metadata>\n"," <defs>\n","  <style type=\"text/css\">*{stroke-linejoin: round; stroke-linecap: butt}</style>\n"," </defs>\n"," <g id=\"figure_1\">\n","  <g id=\"patch_1\">\n","   <path d=\"M 0 166.978125 \n","L 288.403125 166.978125 \n","L 288.403125 0 \n","L 0 0 \n","L 0 166.978125 \n","z\n","\" style=\"fill: none\"/>\n","  </g>\n","  <g id=\"axes_1\">\n","   <g id=\"patch_2\">\n","    <path d=\"M 30.103125 143.1 \n","L 281.203125 143.1 \n","L 281.203125 7.2 \n","L 30.103125 7.2 \n","z\n","\" style=\"fill: #ffffff\"/>\n","   </g>\n","   <g id=\"matplotlib.axis_1\">\n","    <g id=\"xtick_1\">\n","     <g id=\"line2d_1\">\n","      <path d=\"M 48.695149 143.1 \n","L 48.695149 7.2 \n","\" clip-path=\"url(#p66af7e3b3a)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n","     </g>\n","     <g id=\"line2d_2\">\n","      <defs>\n","       <path id=\"me9b6ac727b\" d=\"M 0 0 \n","L 0 3.5 \n","\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n","      </defs>\n","      <g>\n","       <use xlink:href=\"#me9b6ac727b\" x=\"48.695149\" y=\"143.1\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n","      </g>\n","     </g>\n","     <g id=\"text_1\">\n","      <!-- −7.5 -->\n","      <g transform=\"translate(36.553743 157.698438)scale(0.1 -0.1)\">\n","       <defs>\n","        <path id=\"DejaVuSans-2212\" d=\"M 678 2272 \n","L 4684 2272 \n","L 4684 1741 \n","L 678 1741 \n","L 678 2272 \n","z\n","\" transform=\"scale(0.015625)\"/>\n","        <path id=\"DejaVuSans-37\" d=\"M 525 4666 \n","L 3525 4666 \n","L 3525 4397 \n","L 1831 0 \n","L 1172 0 \n","L 2766 4134 \n","L 525 4134 \n","L 525 4666 \n","z\n","\" transform=\"scale(0.015625)\"/>\n","        <path id=\"DejaVuSans-2e\" d=\"M 684 794 \n","L 1344 794 \n","L 1344 0 \n","L 684 0 \n","L 684 794 \n","z\n","\" transform=\"scale(0.015625)\"/>\n","        <path id=\"DejaVuSans-35\" d=\"M 691 4666 \n","L 3169 4666 \n","L 3169 4134 \n","L 1269 4134 \n","L 1269 2991 \n","Q 1406 3038 1543 3061 \n","Q 1681 3084 1819 3084 \n","Q 2600 3084 3056 2656 \n","Q 3513 2228 3513 1497 \n","Q 3513 744 3044 326 \n","Q 2575 -91 1722 -91 \n","Q 1428 -91 1123 -41 \n","Q 819 9 494 109 \n","L 494 744 \n","Q 775 591 1075 516 \n","Q 1375 441 1709 441 \n","Q 2250 441 2565 725 \n","Q 2881 1009 2881 1497 \n","Q 2881 1984 2565 2268 \n","Q 2250 2553 1709 2553 \n","Q 1456 2553 1204 2497 \n","Q 953 2441 691 2322 \n","L 691 4666 \n","z\n","\" transform=\"scale(0.015625)\"/>\n","       </defs>\n","       <use xlink:href=\"#DejaVuSans-2212\"/>\n","       <use xlink:href=\"#DejaVuSans-37\" x=\"83.789062\"/>\n","       <use xlink:href=\"#DejaVuSans-2e\" x=\"147.412109\"/>\n","       <use xlink:href=\"#DejaVuSans-35\" x=\"179.199219\"/>\n","      </g>\n","     </g>\n","    </g>\n","    <g id=\"xtick_2\">\n","     <g id=\"line2d_3\">\n","      <path d=\"M 84.587088 143.1 \n","L 84.587088 7.2 \n","\" clip-path=\"url(#p66af7e3b3a)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n","     </g>\n","     <g id=\"line2d_4\">\n","      <g>\n","       <use xlink:href=\"#me9b6ac727b\" x=\"84.587088\" y=\"143.1\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n","      </g>\n","     </g>\n","     <g id=\"text_2\">\n","      <!-- −5.0 -->\n","      <g transform=\"translate(72.445682 157.698438)scale(0.1 -0.1)\">\n","       <defs>\n","        <path id=\"DejaVuSans-30\" d=\"M 2034 4250 \n","Q 1547 4250 1301 3770 \n","Q 1056 3291 1056 2328 \n","Q 1056 1369 1301 889 \n","Q 1547 409 2034 409 \n","Q 2525 409 2770 889 \n","Q 3016 1369 3016 2328 \n","Q 3016 3291 2770 3770 \n","Q 2525 4250 2034 4250 \n","z\n","M 2034 4750 \n","Q 2819 4750 3233 4129 \n","Q 3647 3509 3647 2328 \n","Q 3647 1150 3233 529 \n","Q 2819 -91 2034 -91 \n","Q 1250 -91 836 529 \n","Q 422 1150 422 2328 \n","Q 422 3509 836 4129 \n","Q 1250 4750 2034 4750 \n","z\n","\" transform=\"scale(0.015625)\"/>\n","       </defs>\n","       <use xlink:href=\"#DejaVuSans-2212\"/>\n","       <use xlink:href=\"#DejaVuSans-35\" x=\"83.789062\"/>\n","       <use xlink:href=\"#DejaVuSans-2e\" x=\"147.412109\"/>\n","       <use xlink:href=\"#DejaVuSans-30\" x=\"179.199219\"/>\n","      </g>\n","     </g>\n","    </g>\n","    <g id=\"xtick_3\">\n","     <g id=\"line2d_5\">\n","      <path d=\"M 120.479027 143.1 \n","L 120.479027 7.2 \n","\" clip-path=\"url(#p66af7e3b3a)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n","     </g>\n","     <g id=\"line2d_6\">\n","      <g>\n","       <use xlink:href=\"#me9b6ac727b\" x=\"120.479027\" y=\"143.1\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n","      </g>\n","     </g>\n","     <g id=\"text_3\">\n","      <!-- −2.5 -->\n","      <g transform=\"translate(108.337621 157.698438)scale(0.1 -0.1)\">\n","       <defs>\n","        <path id=\"DejaVuSans-32\" d=\"M 1228 531 \n","L 3431 531 \n","L 3431 0 \n","L 469 0 \n","L 469 531 \n","Q 828 903 1448 1529 \n","Q 2069 2156 2228 2338 \n","Q 2531 2678 2651 2914 \n","Q 2772 3150 2772 3378 \n","Q 2772 3750 2511 3984 \n","Q 2250 4219 1831 4219 \n","Q 1534 4219 1204 4116 \n","Q 875 4013 500 3803 \n","L 500 4441 \n","Q 881 4594 1212 4672 \n","Q 1544 4750 1819 4750 \n","Q 2544 4750 2975 4387 \n","Q 3406 4025 3406 3419 \n","Q 3406 3131 3298 2873 \n","Q 3191 2616 2906 2266 \n","Q 2828 2175 2409 1742 \n","Q 1991 1309 1228 531 \n","z\n","\" transform=\"scale(0.015625)\"/>\n","       </defs>\n","       <use xlink:href=\"#DejaVuSans-2212\"/>\n","       <use xlink:href=\"#DejaVuSans-32\" x=\"83.789062\"/>\n","       <use xlink:href=\"#DejaVuSans-2e\" x=\"147.412109\"/>\n","       <use xlink:href=\"#DejaVuSans-35\" x=\"179.199219\"/>\n","      </g>\n","     </g>\n","    </g>\n","    <g id=\"xtick_4\">\n","     <g id=\"line2d_7\">\n","      <path d=\"M 156.370967 143.1 \n","L 156.370967 7.2 \n","\" clip-path=\"url(#p66af7e3b3a)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n","     </g>\n","     <g id=\"line2d_8\">\n","      <g>\n","       <use xlink:href=\"#me9b6ac727b\" x=\"156.370967\" y=\"143.1\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n","      </g>\n","     </g>\n","     <g id=\"text_4\">\n","      <!-- 0.0 -->\n","      <g transform=\"translate(148.419404 157.698438)scale(0.1 -0.1)\">\n","       <use xlink:href=\"#DejaVuSans-30\"/>\n","       <use xlink:href=\"#DejaVuSans-2e\" x=\"63.623047\"/>\n","       <use xlink:href=\"#DejaVuSans-30\" x=\"95.410156\"/>\n","      </g>\n","     </g>\n","    </g>\n","    <g id=\"xtick_5\">\n","     <g id=\"line2d_9\">\n","      <path d=\"M 192.262906 143.1 \n","L 192.262906 7.2 \n","\" clip-path=\"url(#p66af7e3b3a)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n","     </g>\n","     <g id=\"line2d_10\">\n","      <g>\n","       <use xlink:href=\"#me9b6ac727b\" x=\"192.262906\" y=\"143.1\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n","      </g>\n","     </g>\n","     <g id=\"text_5\">\n","      <!-- 2.5 -->\n","      <g transform=\"translate(184.311343 157.698438)scale(0.1 -0.1)\">\n","       <use xlink:href=\"#DejaVuSans-32\"/>\n","       <use xlink:href=\"#DejaVuSans-2e\" x=\"63.623047\"/>\n","       <use xlink:href=\"#DejaVuSans-35\" x=\"95.410156\"/>\n","      </g>\n","     </g>\n","    </g>\n","    <g id=\"xtick_6\">\n","     <g id=\"line2d_11\">\n","      <path d=\"M 228.154845 143.1 \n","L 228.154845 7.2 \n","\" clip-path=\"url(#p66af7e3b3a)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n","     </g>\n","     <g id=\"line2d_12\">\n","      <g>\n","       <use xlink:href=\"#me9b6ac727b\" x=\"228.154845\" y=\"143.1\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n","      </g>\n","     </g>\n","     <g id=\"text_6\">\n","      <!-- 5.0 -->\n","      <g transform=\"translate(220.203282 157.698438)scale(0.1 -0.1)\">\n","       <use xlink:href=\"#DejaVuSans-35\"/>\n","       <use xlink:href=\"#DejaVuSans-2e\" x=\"63.623047\"/>\n","       <use xlink:href=\"#DejaVuSans-30\" x=\"95.410156\"/>\n","      </g>\n","     </g>\n","    </g>\n","    <g id=\"xtick_7\">\n","     <g id=\"line2d_13\">\n","      <path d=\"M 264.046784 143.1 \n","L 264.046784 7.2 \n","\" clip-path=\"url(#p66af7e3b3a)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n","     </g>\n","     <g id=\"line2d_14\">\n","      <g>\n","       <use xlink:href=\"#me9b6ac727b\" x=\"264.046784\" y=\"143.1\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n","      </g>\n","     </g>\n","     <g id=\"text_7\">\n","      <!-- 7.5 -->\n","      <g transform=\"translate(256.095221 157.698438)scale(0.1 -0.1)\">\n","       <use xlink:href=\"#DejaVuSans-37\"/>\n","       <use xlink:href=\"#DejaVuSans-2e\" x=\"63.623047\"/>\n","       <use xlink:href=\"#DejaVuSans-35\" x=\"95.410156\"/>\n","      </g>\n","     </g>\n","    </g>\n","   </g>\n","   <g id=\"matplotlib.axis_2\">\n","    <g id=\"ytick_1\">\n","     <g id=\"line2d_15\">\n","      <path d=\"M 30.103125 136.964174 \n","L 281.203125 136.964174 \n","\" clip-path=\"url(#p66af7e3b3a)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n","     </g>\n","     <g id=\"line2d_16\">\n","      <defs>\n","       <path id=\"m5eabb309cf\" d=\"M 0 0 \n","L -3.5 0 \n","\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n","      </defs>\n","      <g>\n","       <use xlink:href=\"#m5eabb309cf\" x=\"30.103125\" y=\"136.964174\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n","      </g>\n","     </g>\n","     <g id=\"text_8\">\n","      <!-- 0.0 -->\n","      <g transform=\"translate(7.2 140.763392)scale(0.1 -0.1)\">\n","       <use xlink:href=\"#DejaVuSans-30\"/>\n","       <use xlink:href=\"#DejaVuSans-2e\" x=\"63.623047\"/>\n","       <use xlink:href=\"#DejaVuSans-30\" x=\"95.410156\"/>\n","      </g>\n","     </g>\n","    </g>\n","    <g id=\"ytick_2\">\n","     <g id=\"line2d_17\">\n","      <path d=\"M 30.103125 112.237629 \n","L 281.203125 112.237629 \n","\" clip-path=\"url(#p66af7e3b3a)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n","     </g>\n","     <g id=\"line2d_18\">\n","      <g>\n","       <use xlink:href=\"#m5eabb309cf\" x=\"30.103125\" y=\"112.237629\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n","      </g>\n","     </g>\n","     <g id=\"text_9\">\n","      <!-- 0.2 -->\n","      <g transform=\"translate(7.2 116.036848)scale(0.1 -0.1)\">\n","       <use xlink:href=\"#DejaVuSans-30\"/>\n","       <use xlink:href=\"#DejaVuSans-2e\" x=\"63.623047\"/>\n","       <use xlink:href=\"#DejaVuSans-32\" x=\"95.410156\"/>\n","      </g>\n","     </g>\n","    </g>\n","    <g id=\"ytick_3\">\n","     <g id=\"line2d_19\">\n","      <path d=\"M 30.103125 87.511085 \n","L 281.203125 87.511085 \n","\" clip-path=\"url(#p66af7e3b3a)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n","     </g>\n","     <g id=\"line2d_20\">\n","      <g>\n","       <use xlink:href=\"#m5eabb309cf\" x=\"30.103125\" y=\"87.511085\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n","      </g>\n","     </g>\n","     <g id=\"text_10\">\n","      <!-- 0.4 -->\n","      <g transform=\"translate(7.2 91.310304)scale(0.1 -0.1)\">\n","       <defs>\n","        <path id=\"DejaVuSans-34\" d=\"M 2419 4116 \n","L 825 1625 \n","L 2419 1625 \n","L 2419 4116 \n","z\n","M 2253 4666 \n","L 3047 4666 \n","L 3047 1625 \n","L 3713 1625 \n","L 3713 1100 \n","L 3047 1100 \n","L 3047 0 \n","L 2419 0 \n","L 2419 1100 \n","L 313 1100 \n","L 313 1709 \n","L 2253 4666 \n","z\n","\" transform=\"scale(0.015625)\"/>\n","       </defs>\n","       <use xlink:href=\"#DejaVuSans-30\"/>\n","       <use xlink:href=\"#DejaVuSans-2e\" x=\"63.623047\"/>\n","       <use xlink:href=\"#DejaVuSans-34\" x=\"95.410156\"/>\n","      </g>\n","     </g>\n","    </g>\n","    <g id=\"ytick_4\">\n","     <g id=\"line2d_21\">\n","      <path d=\"M 30.103125 62.784541 \n","L 281.203125 62.784541 \n","\" clip-path=\"url(#p66af7e3b3a)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n","     </g>\n","     <g id=\"line2d_22\">\n","      <g>\n","       <use xlink:href=\"#m5eabb309cf\" x=\"30.103125\" y=\"62.784541\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n","      </g>\n","     </g>\n","     <g id=\"text_11\">\n","      <!-- 0.6 -->\n","      <g transform=\"translate(7.2 66.583759)scale(0.1 -0.1)\">\n","       <defs>\n","        <path id=\"DejaVuSans-36\" d=\"M 2113 2584 \n","Q 1688 2584 1439 2293 \n","Q 1191 2003 1191 1497 \n","Q 1191 994 1439 701 \n","Q 1688 409 2113 409 \n","Q 2538 409 2786 701 \n","Q 3034 994 3034 1497 \n","Q 3034 2003 2786 2293 \n","Q 2538 2584 2113 2584 \n","z\n","M 3366 4563 \n","L 3366 3988 \n","Q 3128 4100 2886 4159 \n","Q 2644 4219 2406 4219 \n","Q 1781 4219 1451 3797 \n","Q 1122 3375 1075 2522 \n","Q 1259 2794 1537 2939 \n","Q 1816 3084 2150 3084 \n","Q 2853 3084 3261 2657 \n","Q 3669 2231 3669 1497 \n","Q 3669 778 3244 343 \n","Q 2819 -91 2113 -91 \n","Q 1303 -91 875 529 \n","Q 447 1150 447 2328 \n","Q 447 3434 972 4092 \n","Q 1497 4750 2381 4750 \n","Q 2619 4750 2861 4703 \n","Q 3103 4656 3366 4563 \n","z\n","\" transform=\"scale(0.015625)\"/>\n","       </defs>\n","       <use xlink:href=\"#DejaVuSans-30\"/>\n","       <use xlink:href=\"#DejaVuSans-2e\" x=\"63.623047\"/>\n","       <use xlink:href=\"#DejaVuSans-36\" x=\"95.410156\"/>\n","      </g>\n","     </g>\n","    </g>\n","    <g id=\"ytick_5\">\n","     <g id=\"line2d_23\">\n","      <path d=\"M 30.103125 38.057996 \n","L 281.203125 38.057996 \n","\" clip-path=\"url(#p66af7e3b3a)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n","     </g>\n","     <g id=\"line2d_24\">\n","      <g>\n","       <use xlink:href=\"#m5eabb309cf\" x=\"30.103125\" y=\"38.057996\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n","      </g>\n","     </g>\n","     <g id=\"text_12\">\n","      <!-- 0.8 -->\n","      <g transform=\"translate(7.2 41.857215)scale(0.1 -0.1)\">\n","       <defs>\n","        <path id=\"DejaVuSans-38\" d=\"M 2034 2216 \n","Q 1584 2216 1326 1975 \n","Q 1069 1734 1069 1313 \n","Q 1069 891 1326 650 \n","Q 1584 409 2034 409 \n","Q 2484 409 2743 651 \n","Q 3003 894 3003 1313 \n","Q 3003 1734 2745 1975 \n","Q 2488 2216 2034 2216 \n","z\n","M 1403 2484 \n","Q 997 2584 770 2862 \n","Q 544 3141 544 3541 \n","Q 544 4100 942 4425 \n","Q 1341 4750 2034 4750 \n","Q 2731 4750 3128 4425 \n","Q 3525 4100 3525 3541 \n","Q 3525 3141 3298 2862 \n","Q 3072 2584 2669 2484 \n","Q 3125 2378 3379 2068 \n","Q 3634 1759 3634 1313 \n","Q 3634 634 3220 271 \n","Q 2806 -91 2034 -91 \n","Q 1263 -91 848 271 \n","Q 434 634 434 1313 \n","Q 434 1759 690 2068 \n","Q 947 2378 1403 2484 \n","z\n","M 1172 3481 \n","Q 1172 3119 1398 2916 \n","Q 1625 2713 2034 2713 \n","Q 2441 2713 2670 2916 \n","Q 2900 3119 2900 3481 \n","Q 2900 3844 2670 4047 \n","Q 2441 4250 2034 4250 \n","Q 1625 4250 1398 4047 \n","Q 1172 3844 1172 3481 \n","z\n","\" transform=\"scale(0.015625)\"/>\n","       </defs>\n","       <use xlink:href=\"#DejaVuSans-30\"/>\n","       <use xlink:href=\"#DejaVuSans-2e\" x=\"63.623047\"/>\n","       <use xlink:href=\"#DejaVuSans-38\" x=\"95.410156\"/>\n","      </g>\n","     </g>\n","    </g>\n","    <g id=\"ytick_6\">\n","     <g id=\"line2d_25\">\n","      <path d=\"M 30.103125 13.331452 \n","L 281.203125 13.331452 \n","\" clip-path=\"url(#p66af7e3b3a)\" style=\"fill: none; stroke: #b0b0b0; stroke-width: 0.8; stroke-linecap: square\"/>\n","     </g>\n","     <g id=\"line2d_26\">\n","      <g>\n","       <use xlink:href=\"#m5eabb309cf\" x=\"30.103125\" y=\"13.331452\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n","      </g>\n","     </g>\n","     <g id=\"text_13\">\n","      <!-- 1.0 -->\n","      <g transform=\"translate(7.2 17.130671)scale(0.1 -0.1)\">\n","       <defs>\n","        <path id=\"DejaVuSans-31\" d=\"M 794 531 \n","L 1825 531 \n","L 1825 4091 \n","L 703 3866 \n","L 703 4441 \n","L 1819 4666 \n","L 2450 4666 \n","L 2450 531 \n","L 3481 531 \n","L 3481 0 \n","L 794 0 \n","L 794 531 \n","z\n","\" transform=\"scale(0.015625)\"/>\n","       </defs>\n","       <use xlink:href=\"#DejaVuSans-31\"/>\n","       <use xlink:href=\"#DejaVuSans-2e\" x=\"63.623047\"/>\n","       <use xlink:href=\"#DejaVuSans-30\" x=\"95.410156\"/>\n","      </g>\n","     </g>\n","    </g>\n","   </g>\n","   <g id=\"line2d_27\">\n","    <path d=\"M 41.516761 136.922713 \n","L 65.923277 136.737562 \n","L 77.4087 136.460971 \n","L 86.022764 136.050337 \n","L 93.201152 135.464702 \n","L 98.943864 134.74049 \n","L 103.250896 133.981285 \n","L 107.557928 132.971398 \n","L 110.429284 132.122009 \n","L 113.30064 131.100784 \n","L 116.171995 129.87703 \n","L 119.043348 128.416404 \n","L 121.914704 126.681307 \n","L 124.786059 124.63175 \n","L 127.657415 122.226792 \n","L 130.528769 119.426742 \n","L 133.400125 116.19615 \n","L 136.271481 112.50763 \n","L 139.142835 108.346262 \n","L 142.014191 103.714212 \n","L 144.885546 98.634876 \n","L 147.756901 93.155696 \n","L 152.063934 84.351342 \n","L 164.985032 57.139929 \n","L 167.856387 51.66075 \n","L 170.727742 46.581409 \n","L 173.599098 41.949358 \n","L 176.470452 37.787991 \n","L 179.341808 34.099477 \n","L 182.213164 30.868893 \n","L 185.084518 28.06884 \n","L 187.955874 25.663873 \n","L 190.827229 23.614316 \n","L 193.698585 21.879221 \n","L 196.569941 20.418595 \n","L 199.441293 19.194841 \n","L 202.312649 18.173618 \n","L 206.619681 16.955405 \n","L 210.926713 16.036702 \n","L 215.233745 15.346977 \n","L 220.976457 14.689795 \n","L 228.154845 14.158904 \n","L 236.768909 13.786942 \n","L 248.254332 13.536533 \n","L 266.918136 13.38742 \n","L 269.789489 13.377273 \n","L 269.789489 13.377273 \n","\" clip-path=\"url(#p66af7e3b3a)\" style=\"fill: none; stroke: #1f77b4; stroke-width: 1.5; stroke-linecap: square\"/>\n","   </g>\n","   <g id=\"line2d_28\">\n","    <path d=\"M 41.516761 136.922727 \n","L 65.923277 136.737978 \n","L 77.4087 136.46302 \n","L 86.022764 136.057092 \n","L 93.201152 135.482889 \n","L 98.943864 134.780485 \n","L 103.250896 134.053253 \n","L 107.557928 133.100346 \n","L 111.86496 131.864446 \n","L 114.736316 130.852567 \n","L 117.607672 129.66889 \n","L 120.479027 128.297061 \n","L 123.35038 126.724966 \n","L 126.221736 124.947729 \n","L 129.093091 122.971391 \n","L 133.400125 119.684796 \n","L 144.885546 110.517935 \n","L 147.756901 108.678954 \n","L 150.628256 107.260083 \n","L 152.063934 106.741126 \n","L 153.499611 106.363024 \n","L 154.935289 106.133133 \n","L 156.370967 106.055993 \n","L 157.806644 106.133135 \n","L 159.242322 106.363026 \n","L 160.677999 106.741126 \n","L 162.113677 107.260083 \n","L 164.985032 108.678954 \n","L 167.856387 110.517935 \n","L 170.727742 112.656506 \n","L 176.470452 117.34553 \n","L 180.777486 120.817036 \n","L 185.084518 123.983526 \n","L 187.955874 125.861917 \n","L 190.827229 127.536562 \n","L 193.698585 129.007383 \n","L 196.569941 130.283295 \n","L 199.441293 131.378861 \n","L 203.748325 132.722848 \n","L 208.055361 133.763369 \n","L 212.362393 134.559937 \n","L 218.105105 135.33127 \n","L 223.847809 135.859872 \n","L 231.026204 136.289615 \n","L 241.075944 136.627342 \n","L 255.43272 136.839828 \n","L 269.789489 136.91837 \n","L 269.789489 136.91837 \n","\" clip-path=\"url(#p66af7e3b3a)\" style=\"fill: none; stroke-dasharray: 5.55,2.4; stroke-dashoffset: 0; stroke: #bf00bf; stroke-width: 1.5\"/>\n","   </g>\n","   <g id=\"patch_3\">\n","    <path d=\"M 30.103125 143.1 \n","L 30.103125 7.2 \n","\" style=\"fill: none; stroke: #000000; stroke-width: 0.8; stroke-linejoin: miter; stroke-linecap: square\"/>\n","   </g>\n","   <g id=\"patch_4\">\n","    <path d=\"M 281.203125 143.1 \n","L 281.203125 7.2 \n","\" style=\"fill: none; stroke: #000000; stroke-width: 0.8; stroke-linejoin: miter; stroke-linecap: square\"/>\n","   </g>\n","   <g id=\"patch_5\">\n","    <path d=\"M 30.103125 143.1 \n","L 281.203125 143.1 \n","\" style=\"fill: none; stroke: #000000; stroke-width: 0.8; stroke-linejoin: miter; stroke-linecap: square\"/>\n","   </g>\n","   <g id=\"patch_6\">\n","    <path d=\"M 30.103125 7.2 \n","L 281.203125 7.2 \n","\" style=\"fill: none; stroke: #000000; stroke-width: 0.8; stroke-linejoin: miter; stroke-linecap: square\"/>\n","   </g>\n","   <g id=\"legend_1\">\n","    <g id=\"patch_7\">\n","     <path d=\"M 37.103125 44.55625 \n","L 111.228125 44.55625 \n","Q 113.228125 44.55625 113.228125 42.55625 \n","L 113.228125 14.2 \n","Q 113.228125 12.2 111.228125 12.2 \n","L 37.103125 12.2 \n","Q 35.103125 12.2 35.103125 14.2 \n","L 35.103125 42.55625 \n","Q 35.103125 44.55625 37.103125 44.55625 \n","z\n","\" style=\"fill: #ffffff; opacity: 0.8; stroke: #cccccc; stroke-linejoin: miter\"/>\n","    </g>\n","    <g id=\"line2d_29\">\n","     <path d=\"M 39.103125 20.298438 \n","L 49.103125 20.298438 \n","L 59.103125 20.298438 \n","\" style=\"fill: none; stroke: #1f77b4; stroke-width: 1.5; stroke-linecap: square\"/>\n","    </g>\n","    <g id=\"text_14\">\n","     <!-- sigmoid -->\n","     <g transform=\"translate(67.103125 23.798438)scale(0.1 -0.1)\">\n","      <defs>\n","       <path id=\"DejaVuSans-73\" d=\"M 2834 3397 \n","L 2834 2853 \n","Q 2591 2978 2328 3040 \n","Q 2066 3103 1784 3103 \n","Q 1356 3103 1142 2972 \n","Q 928 2841 928 2578 \n","Q 928 2378 1081 2264 \n","Q 1234 2150 1697 2047 \n","L 1894 2003 \n","Q 2506 1872 2764 1633 \n","Q 3022 1394 3022 966 \n","Q 3022 478 2636 193 \n","Q 2250 -91 1575 -91 \n","Q 1294 -91 989 -36 \n","Q 684 19 347 128 \n","L 347 722 \n","Q 666 556 975 473 \n","Q 1284 391 1588 391 \n","Q 1994 391 2212 530 \n","Q 2431 669 2431 922 \n","Q 2431 1156 2273 1281 \n","Q 2116 1406 1581 1522 \n","L 1381 1569 \n","Q 847 1681 609 1914 \n","Q 372 2147 372 2553 \n","Q 372 3047 722 3315 \n","Q 1072 3584 1716 3584 \n","Q 2034 3584 2315 3537 \n","Q 2597 3491 2834 3397 \n","z\n","\" transform=\"scale(0.015625)\"/>\n","       <path id=\"DejaVuSans-69\" d=\"M 603 3500 \n","L 1178 3500 \n","L 1178 0 \n","L 603 0 \n","L 603 3500 \n","z\n","M 603 4863 \n","L 1178 4863 \n","L 1178 4134 \n","L 603 4134 \n","L 603 4863 \n","z\n","\" transform=\"scale(0.015625)\"/>\n","       <path id=\"DejaVuSans-67\" d=\"M 2906 1791 \n","Q 2906 2416 2648 2759 \n","Q 2391 3103 1925 3103 \n","Q 1463 3103 1205 2759 \n","Q 947 2416 947 1791 \n","Q 947 1169 1205 825 \n","Q 1463 481 1925 481 \n","Q 2391 481 2648 825 \n","Q 2906 1169 2906 1791 \n","z\n","M 3481 434 \n","Q 3481 -459 3084 -895 \n","Q 2688 -1331 1869 -1331 \n","Q 1566 -1331 1297 -1286 \n","Q 1028 -1241 775 -1147 \n","L 775 -588 \n","Q 1028 -725 1275 -790 \n","Q 1522 -856 1778 -856 \n","Q 2344 -856 2625 -561 \n","Q 2906 -266 2906 331 \n","L 2906 616 \n","Q 2728 306 2450 153 \n","Q 2172 0 1784 0 \n","Q 1141 0 747 490 \n","Q 353 981 353 1791 \n","Q 353 2603 747 3093 \n","Q 1141 3584 1784 3584 \n","Q 2172 3584 2450 3431 \n","Q 2728 3278 2906 2969 \n","L 2906 3500 \n","L 3481 3500 \n","L 3481 434 \n","z\n","\" transform=\"scale(0.015625)\"/>\n","       <path id=\"DejaVuSans-6d\" d=\"M 3328 2828 \n","Q 3544 3216 3844 3400 \n","Q 4144 3584 4550 3584 \n","Q 5097 3584 5394 3201 \n","Q 5691 2819 5691 2113 \n","L 5691 0 \n","L 5113 0 \n","L 5113 2094 \n","Q 5113 2597 4934 2840 \n","Q 4756 3084 4391 3084 \n","Q 3944 3084 3684 2787 \n","Q 3425 2491 3425 1978 \n","L 3425 0 \n","L 2847 0 \n","L 2847 2094 \n","Q 2847 2600 2669 2842 \n","Q 2491 3084 2119 3084 \n","Q 1678 3084 1418 2786 \n","Q 1159 2488 1159 1978 \n","L 1159 0 \n","L 581 0 \n","L 581 3500 \n","L 1159 3500 \n","L 1159 2956 \n","Q 1356 3278 1631 3431 \n","Q 1906 3584 2284 3584 \n","Q 2666 3584 2933 3390 \n","Q 3200 3197 3328 2828 \n","z\n","\" transform=\"scale(0.015625)\"/>\n","       <path id=\"DejaVuSans-6f\" d=\"M 1959 3097 \n","Q 1497 3097 1228 2736 \n","Q 959 2375 959 1747 \n","Q 959 1119 1226 758 \n","Q 1494 397 1959 397 \n","Q 2419 397 2687 759 \n","Q 2956 1122 2956 1747 \n","Q 2956 2369 2687 2733 \n","Q 2419 3097 1959 3097 \n","z\n","M 1959 3584 \n","Q 2709 3584 3137 3096 \n","Q 3566 2609 3566 1747 \n","Q 3566 888 3137 398 \n","Q 2709 -91 1959 -91 \n","Q 1206 -91 779 398 \n","Q 353 888 353 1747 \n","Q 353 2609 779 3096 \n","Q 1206 3584 1959 3584 \n","z\n","\" transform=\"scale(0.015625)\"/>\n","       <path id=\"DejaVuSans-64\" d=\"M 2906 2969 \n","L 2906 4863 \n","L 3481 4863 \n","L 3481 0 \n","L 2906 0 \n","L 2906 525 \n","Q 2725 213 2448 61 \n","Q 2172 -91 1784 -91 \n","Q 1150 -91 751 415 \n","Q 353 922 353 1747 \n","Q 353 2572 751 3078 \n","Q 1150 3584 1784 3584 \n","Q 2172 3584 2448 3432 \n","Q 2725 3281 2906 2969 \n","z\n","M 947 1747 \n","Q 947 1113 1208 752 \n","Q 1469 391 1925 391 \n","Q 2381 391 2643 752 \n","Q 2906 1113 2906 1747 \n","Q 2906 2381 2643 2742 \n","Q 2381 3103 1925 3103 \n","Q 1469 3103 1208 2742 \n","Q 947 2381 947 1747 \n","z\n","\" transform=\"scale(0.015625)\"/>\n","      </defs>\n","      <use xlink:href=\"#DejaVuSans-73\"/>\n","      <use xlink:href=\"#DejaVuSans-69\" x=\"52.099609\"/>\n","      <use xlink:href=\"#DejaVuSans-67\" x=\"79.882812\"/>\n","      <use xlink:href=\"#DejaVuSans-6d\" x=\"143.359375\"/>\n","      <use xlink:href=\"#DejaVuSans-6f\" x=\"240.771484\"/>\n","      <use xlink:href=\"#DejaVuSans-69\" x=\"301.953125\"/>\n","      <use xlink:href=\"#DejaVuSans-64\" x=\"329.736328\"/>\n","     </g>\n","    </g>\n","    <g id=\"line2d_30\">\n","     <path d=\"M 39.103125 34.976562 \n","L 49.103125 34.976562 \n","L 59.103125 34.976562 \n","\" style=\"fill: none; stroke-dasharray: 5.55,2.4; stroke-dashoffset: 0; stroke: #bf00bf; stroke-width: 1.5\"/>\n","    </g>\n","    <g id=\"text_15\">\n","     <!-- gradient -->\n","     <g transform=\"translate(67.103125 38.476562)scale(0.1 -0.1)\">\n","      <defs>\n","       <path id=\"DejaVuSans-72\" d=\"M 2631 2963 \n","Q 2534 3019 2420 3045 \n","Q 2306 3072 2169 3072 \n","Q 1681 3072 1420 2755 \n","Q 1159 2438 1159 1844 \n","L 1159 0 \n","L 581 0 \n","L 581 3500 \n","L 1159 3500 \n","L 1159 2956 \n","Q 1341 3275 1631 3429 \n","Q 1922 3584 2338 3584 \n","Q 2397 3584 2469 3576 \n","Q 2541 3569 2628 3553 \n","L 2631 2963 \n","z\n","\" transform=\"scale(0.015625)\"/>\n","       <path id=\"DejaVuSans-61\" d=\"M 2194 1759 \n","Q 1497 1759 1228 1600 \n","Q 959 1441 959 1056 \n","Q 959 750 1161 570 \n","Q 1363 391 1709 391 \n","Q 2188 391 2477 730 \n","Q 2766 1069 2766 1631 \n","L 2766 1759 \n","L 2194 1759 \n","z\n","M 3341 1997 \n","L 3341 0 \n","L 2766 0 \n","L 2766 531 \n","Q 2569 213 2275 61 \n","Q 1981 -91 1556 -91 \n","Q 1019 -91 701 211 \n","Q 384 513 384 1019 \n","Q 384 1609 779 1909 \n","Q 1175 2209 1959 2209 \n","L 2766 2209 \n","L 2766 2266 \n","Q 2766 2663 2505 2880 \n","Q 2244 3097 1772 3097 \n","Q 1472 3097 1187 3025 \n","Q 903 2953 641 2809 \n","L 641 3341 \n","Q 956 3463 1253 3523 \n","Q 1550 3584 1831 3584 \n","Q 2591 3584 2966 3190 \n","Q 3341 2797 3341 1997 \n","z\n","\" transform=\"scale(0.015625)\"/>\n","       <path id=\"DejaVuSans-65\" d=\"M 3597 1894 \n","L 3597 1613 \n","L 953 1613 \n","Q 991 1019 1311 708 \n","Q 1631 397 2203 397 \n","Q 2534 397 2845 478 \n","Q 3156 559 3463 722 \n","L 3463 178 \n","Q 3153 47 2828 -22 \n","Q 2503 -91 2169 -91 \n","Q 1331 -91 842 396 \n","Q 353 884 353 1716 \n","Q 353 2575 817 3079 \n","Q 1281 3584 2069 3584 \n","Q 2775 3584 3186 3129 \n","Q 3597 2675 3597 1894 \n","z\n","M 3022 2063 \n","Q 3016 2534 2758 2815 \n","Q 2500 3097 2075 3097 \n","Q 1594 3097 1305 2825 \n","Q 1016 2553 972 2059 \n","L 3022 2063 \n","z\n","\" transform=\"scale(0.015625)\"/>\n","       <path id=\"DejaVuSans-6e\" d=\"M 3513 2113 \n","L 3513 0 \n","L 2938 0 \n","L 2938 2094 \n","Q 2938 2591 2744 2837 \n","Q 2550 3084 2163 3084 \n","Q 1697 3084 1428 2787 \n","Q 1159 2491 1159 1978 \n","L 1159 0 \n","L 581 0 \n","L 581 3500 \n","L 1159 3500 \n","L 1159 2956 \n","Q 1366 3272 1645 3428 \n","Q 1925 3584 2291 3584 \n","Q 2894 3584 3203 3211 \n","Q 3513 2838 3513 2113 \n","z\n","\" transform=\"scale(0.015625)\"/>\n","       <path id=\"DejaVuSans-74\" d=\"M 1172 4494 \n","L 1172 3500 \n","L 2356 3500 \n","L 2356 3053 \n","L 1172 3053 \n","L 1172 1153 \n","Q 1172 725 1289 603 \n","Q 1406 481 1766 481 \n","L 2356 481 \n","L 2356 0 \n","L 1766 0 \n","Q 1100 0 847 248 \n","Q 594 497 594 1153 \n","L 594 3053 \n","L 172 3053 \n","L 172 3500 \n","L 594 3500 \n","L 594 4494 \n","L 1172 4494 \n","z\n","\" transform=\"scale(0.015625)\"/>\n","      </defs>\n","      <use xlink:href=\"#DejaVuSans-67\"/>\n","      <use xlink:href=\"#DejaVuSans-72\" x=\"63.476562\"/>\n","      <use xlink:href=\"#DejaVuSans-61\" x=\"104.589844\"/>\n","      <use xlink:href=\"#DejaVuSans-64\" x=\"165.869141\"/>\n","      <use xlink:href=\"#DejaVuSans-69\" x=\"229.345703\"/>\n","      <use xlink:href=\"#DejaVuSans-65\" x=\"257.128906\"/>\n","      <use xlink:href=\"#DejaVuSans-6e\" x=\"318.652344\"/>\n","      <use xlink:href=\"#DejaVuSans-74\" x=\"382.03125\"/>\n","     </g>\n","    </g>\n","   </g>\n","  </g>\n"," </g>\n"," <defs>\n","  <clipPath id=\"p66af7e3b3a\">\n","   <rect x=\"30.103125\" y=\"7.2\" width=\"251.1\" height=\"135.9\"/>\n","  </clipPath>\n"," </defs>\n","</svg>\n"],"text/plain":["<Figure size 324x180 with 1 Axes>"]},"metadata":{"needs_background":"light"},"output_type":"display_data"}],"source":["%matplotlib inline\n","import torch\n","from d2l import torch as d2l\n","\n","x = torch.arange(-8.0, 8.0, 0.1, requires_grad=True)\n","y = torch.sigmoid(x)\n","y.backward(torch.ones_like(x))\n","\n","d2l.plot(x.detach().numpy(), [y.detach().numpy(), x.grad.numpy()],\n","         legend=['sigmoid', 'gradient'], figsize=(4.5, 2.5))"],"id":"1555bcbf"},{"cell_type":"markdown","metadata":{"origin_pos":4,"id":"2bef0208"},"source":["正如你所看到的，当sigmoid函数的输入很大或是很小时，它的梯度都会消失。\n","此外，当反向传播通过许多层时，除非我们在刚刚好的地方，\n","这些地方sigmoid函数的输入接近于零，否则整个乘积的梯度可能会消失。\n","当我们的网络有很多层时，除非我们很小心，否则在某一层可能会切断梯度。\n","事实上，这个问题曾经困扰着深度网络的训练。\n","因此，更稳定的ReLU系列函数已经成为从业者的默认选择（虽然在神经科学的角度看起来不太合理）。\n","\n","### [**梯度爆炸**]\n","\n","相反，梯度爆炸可能同样令人烦恼。\n","为了更好地说明这一点，我们生成100个高斯随机矩阵，并将它们与某个初始矩阵相乘。\n","对于我们选择的尺度（方差$\\sigma^2=1$），矩阵乘积发生爆炸。\n","当这种情况是由于深度网络的初始化所导致时，我们没有机会让梯度下降优化器收敛。\n"],"id":"2bef0208"},{"cell_type":"code","execution_count":null,"metadata":{"execution":{"iopub.execute_input":"2022-07-31T02:43:29.095921Z","iopub.status.busy":"2022-07-31T02:43:29.095418Z","iopub.status.idle":"2022-07-31T02:43:29.104144Z","shell.execute_reply":"2022-07-31T02:43:29.103237Z"},"origin_pos":6,"tab":["pytorch"],"id":"c5157122","outputId":"fa5523fb-a7e7-491b-8ca2-982c5c721544"},"outputs":[{"name":"stdout","output_type":"stream","text":["一个矩阵 \n"," tensor([[ 0.4382, -0.7687,  0.2731, -0.2587],\n","        [-0.1789, -0.2395,  1.4915,  0.2634],\n","        [-0.5272,  0.2403,  2.4397, -0.7587],\n","        [ 0.9805,  0.4166, -0.1906, -0.2581]])\n","乘以100个矩阵后\n"," tensor([[ 7.6616e+22,  4.2587e+22, -5.8065e+22,  1.2980e+23],\n","        [-2.3790e+21, -1.3224e+21,  1.8030e+21, -4.0304e+21],\n","        [-1.3796e+23, -7.6687e+22,  1.0456e+23, -2.3373e+23],\n","        [ 8.5987e+20,  4.7795e+20, -6.5167e+20,  1.4567e+21]])\n"]}],"source":["M = torch.normal(0, 1, size=(4,4))\n","print('一个矩阵 \\n',M)\n","for i in range(100):\n","    M = torch.mm(M,torch.normal(0, 1, size=(4, 4)))\n","\n","print('乘以100个矩阵后\\n', M)"],"id":"c5157122"},{"cell_type":"markdown","metadata":{"origin_pos":8,"id":"b44b5a5d"},"source":["### 打破对称性\n","\n","神经网络设计中的另一个问题是其参数化所固有的对称性。\n","假设我们有一个简单的多层感知机，它有一个隐藏层和两个隐藏单元。\n","在这种情况下，我们可以对第一层的权重$\\mathbf{W}^{(1)}$进行重排列，\n","并且同样对输出层的权重进行重排列，可以获得相同的函数。\n","第一个隐藏单元与第二个隐藏单元没有什么特别的区别。\n","换句话说，我们在每一层的隐藏单元之间具有排列对称性。\n","\n","假设输出层将上述两个隐藏单元的多层感知机转换为仅一个输出单元。\n","想象一下，如果我们将隐藏层的所有参数初始化为$\\mathbf{W}^{(1)} = c$，\n","$c$为常量，会发生什么？\n","在这种情况下，在前向传播期间，两个隐藏单元采用相同的输入和参数，\n","产生相同的激活，该激活被送到输出单元。\n","在反向传播期间，根据参数$\\mathbf{W}^{(1)}$对输出单元进行微分，\n","得到一个梯度，其元素都取相同的值。\n","因此，在基于梯度的迭代（例如，小批量随机梯度下降）之后，\n","$\\mathbf{W}^{(1)}$的所有元素仍然采用相同的值。\n","这样的迭代永远不会打破对称性，我们可能永远也无法实现网络的表达能力。\n","隐藏层的行为就好像只有一个单元。\n","请注意，虽然小批量随机梯度下降不会打破这种对称性，但暂退法正则化可以。\n","\n","## 参数初始化\n","\n","解决（或至少减轻）上述问题的一种方法是进行参数初始化，\n","优化期间的注意和适当的正则化也可以进一步提高稳定性。\n","\n","### 默认初始化\n","\n","在前面的部分中，例如在 :numref:`sec_linear_concise`中，\n","我们使用正态分布来初始化权重值。如果我们不指定初始化方法，\n","框架将使用默认的随机初始化方法，对于中等难度的问题，这种方法通常很有效。\n","\n","### Xavier初始化\n",":label:`subsec_xavier`\n","\n","让我们看看某些*没有非线性*的全连接层输出（例如，隐藏变量）$o_{i}$的尺度分布。\n","对于该层$n_\\mathrm{in}$输入$x_j$及其相关权重$w_{ij}$，输出由下式给出\n","\n","$$o_{i} = \\sum_{j=1}^{n_\\mathrm{in}} w_{ij} x_j.$$\n","\n","权重$w_{ij}$都是从同一分布中独立抽取的。\n","此外，让我们假设该分布具有零均值和方差$\\sigma^2$。\n","请注意，这并不意味着分布必须是高斯的，只是均值和方差需要存在。\n","现在，让我们假设层$x_j$的输入也具有零均值和方差$\\gamma^2$，\n","并且它们独立于$w_{ij}$并且彼此独立。\n","在这种情况下，我们可以按如下方式计算$o_i$的平均值和方差：\n","\n","$$\n","\\begin{aligned}\n","    E[o_i] & = \\sum_{j=1}^{n_\\mathrm{in}} E[w_{ij} x_j] \\\\&= \\sum_{j=1}^{n_\\mathrm{in}} E[w_{ij}] E[x_j] \\\\&= 0, \\\\\n","    \\mathrm{Var}[o_i] & = E[o_i^2] - (E[o_i])^2 \\\\\n","        & = \\sum_{j=1}^{n_\\mathrm{in}} E[w^2_{ij} x^2_j] - 0 \\\\\n","        & = \\sum_{j=1}^{n_\\mathrm{in}} E[w^2_{ij}] E[x^2_j] \\\\\n","        & = n_\\mathrm{in} \\sigma^2 \\gamma^2.\n","\\end{aligned}\n","$$\n","\n","保持方差不变的一种方法是设置$n_\\mathrm{in} \\sigma^2 = 1$。\n","现在考虑反向传播过程，我们面临着类似的问题，尽管梯度是从更靠近输出的层传播的。\n","使用与前向传播相同的推断，我们可以看到，除非$n_\\mathrm{out} \\sigma^2 = 1$，\n","否则梯度的方差可能会增大，其中$n_\\mathrm{out}$是该层的输出的数量。\n","这使得我们进退两难：我们不可能同时满足这两个条件。\n","相反，我们只需满足：\n","\n","$$\n","\\begin{aligned}\n","\\frac{1}{2} (n_\\mathrm{in} + n_\\mathrm{out}) \\sigma^2 = 1 \\text{ 或等价于 }\n","\\sigma = \\sqrt{\\frac{2}{n_\\mathrm{in} + n_\\mathrm{out}}}.\n","\\end{aligned}\n","$$\n","\n","这就是现在标准且实用的*Xavier初始化*的基础，\n","它以其提出者 :cite:`Glorot.Bengio.2010` 第一作者的名字命名。\n","通常，Xavier初始化从均值为零，方差\n","$\\sigma^2 = \\frac{2}{n_\\mathrm{in} + n_\\mathrm{out}}$\n","的高斯分布中采样权重。\n","我们也可以利用Xavier的直觉来选择从均匀分布中抽取权重时的方差。\n","注意均匀分布$U(-a, a)$的方差为$\\frac{a^2}{3}$。\n","将$\\frac{a^2}{3}$代入到$\\sigma^2$的条件中，将得到初始化值域：\n","\n","$$U\\left(-\\sqrt{\\frac{6}{n_\\mathrm{in} + n_\\mathrm{out}}}, \\sqrt{\\frac{6}{n_\\mathrm{in} + n_\\mathrm{out}}}\\right).$$\n","\n","尽管在上述数学推理中，“不存在非线性”的假设在神经网络中很容易被违反，\n","但Xavier初始化方法在实践中被证明是有效的。\n","\n","### 额外阅读\n","\n","上面的推理仅仅触及了现代参数初始化方法的皮毛。\n","深度学习框架通常实现十几种不同的启发式方法。\n","此外，参数初始化一直是深度学习基础研究的热点领域。\n","其中包括专门用于参数绑定（共享）、超分辨率、序列模型和其他情况的启发式算法。\n","例如，Xiao等人演示了通过使用精心设计的初始化方法\n"," :cite:`Xiao.Bahri.Sohl-Dickstein.ea.2018`，\n","可以无须架构上的技巧而训练10000层神经网络的可能性。\n","\n","如果你对该主题感兴趣，我们建议你深入研究本模块的内容，\n","阅读提出并分析每种启发式方法的论文，然后探索有关该主题的最新出版物。\n","也许你会偶然发现甚至发明一个聪明的想法，并为深度学习框架提供一个实现。\n","\n","## 小结\n","\n","* 梯度消失和梯度爆炸是深度网络中常见的问题。在参数初始化时需要非常小心，以确保梯度和参数可以得到很好的控制。\n","* 需要用启发式的初始化方法来确保初始梯度既不太大也不太小。\n","* ReLU激活函数缓解了梯度消失问题，这样可以加速收敛。\n","* 随机初始化是保证在进行优化前打破对称性的关键。\n","* Xavier初始化表明，对于每一层，输出的方差不受输入数量的影响，任何梯度的方差不受输出数量的影响。\n","\n","## 练习\n","\n","1. 除了多层感知机的排列对称性之外，你能设计出其他神经网络可能会表现出对称性且需要被打破的情况吗？\n","2. 我们是否可以将线性回归或softmax回归中的所有权重参数初始化为相同的值？\n","3. 在相关资料中查找两个矩阵乘积特征值的解析界。这对确保梯度条件合适有什么启示？\n","4. 如果我们知道某些项是发散的，我们能在事后修正吗？看看关于按层自适应速率缩放的论文 :cite:`You.Gitman.Ginsburg.2017` 。\n"],"id":"b44b5a5d"},{"cell_type":"markdown","metadata":{"origin_pos":10,"tab":["pytorch"],"id":"e77508b2"},"source":["[Discussions](https://discuss.d2l.ai/t/1818)\n"],"id":"e77508b2"}],"metadata":{"kernelspec":{"display_name":"Python 3","name":"python3"},"language_info":{"name":"python"},"colab":{"provenance":[{"file_id":"https://github.com/d2l-ai/d2l-zh-pytorch-colab/blob/master/chapter_multilayer-perceptrons/mlp.ipynb","timestamp":1663658129450}],"toc_visible":true,"collapsed_sections":[]}},"nbformat":4,"nbformat_minor":5}